+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20190927/B1a --output-run DEEP/20190927/B1a/science#step1/20240313T094419Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 2.1603 seconds; current memory usage: 0.178 Gibyte, delta: 0.041 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0091 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0224 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/pipeline_orig.qgraph DEEP/20190927/B1a/science#step1/20240313T094419Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 5.9335 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 16.3179 seconds; current memory usage: 0.377 Gibyte, delta: 0.199 Gibyte, peak delta: 0.186 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.377 Gibyte (main), 0.366 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0263 seconds; current memory usage: 0.377 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.377 Gibyte (main), 0.366 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 1116 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20190927_B1a_science#step1_20240313T094419Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.4184 seconds; current memory usage: 0.377 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.377 Gibyte (main), 0.366 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 1118 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.3732 seconds; current memory usage: 0.379 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.379 Gibyte (main), 0.366 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 20229a8a-ab50-41bc-8dcc-00b6e8851a6d parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-13 09:45:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-13 09:45:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 09:45:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-13 09:45:08 proc_lsst.multi:170 [INFO] [multi] job_id 60730 proc_lsst.multi INFO: [multi] job_id 60730 2024-03-13 09:45:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:08 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 60730 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 60730 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 60730 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash dddd78638ba754556c07704f10a6c554 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5847c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/ebd4a761-135e-4742-b637-b43068a9c5b3_isr_898798_15.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/ebd4a761-135e-4742-b637-b43068a9c5b3_isr_898798_15.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App characterizeImage, waiting on task 0 parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App calibrate, waiting on task 1 parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 2 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 4721ad1d15bb1e4f21fc9b868b0624eb parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d585080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/b2e06311-10d8-4985-8c8a-3214df9703bf_isr_898696_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/b2e06311-10d8-4985-8c8a-3214df9703bf_isr_898696_1.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App characterizeImage, waiting on task 3 parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 4 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App calibrate, waiting on task 4 parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 5 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash fc58fc73984aac1864486ae6e7dc56a4 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d585300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/071f84b1-1478-4c33-8db2-7430d66dce7b_isr_898696_36.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/071f84b1-1478-4c33-8db2-7430d66dce7b_isr_898696_36.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App characterizeImage, waiting on task 6 parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 7 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App calibrate, waiting on task 7 parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 8 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 8d7b8fb4370c1e0944ffce5748432410 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5854e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/fb102970-4209-47c8-adc0-797e4a3389a8_isr_898798_6.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/fb102970-4209-47c8-adc0-797e4a3389a8_isr_898798_6.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App characterizeImage, waiting on task 9 parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 10 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App calibrate, waiting on task 10 parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 11 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 72b8d074b5ca9ae9dd1be881d87422ae parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5859e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/03a1391e-5233-44a9-8953-c7988dcebae9_isr_898694_39.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/03a1391e-5233-44a9-8953-c7988dcebae9_isr_898694_39.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App characterizeImage, waiting on task 12 parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 13 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App calibrate, waiting on task 13 parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 14 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 27110a3eb26df4f7dc63a668397508ed parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d586020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/22ba94da-525a-4ef8-be40-7d3f703d2069_isr_898798_57.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/22ba94da-525a-4ef8-be40-7d3f703d2069_isr_898798_57.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App characterizeImage, waiting on task 15 parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 16 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App calibrate, waiting on task 16 parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 17 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash a6f9c6803c6f2e41ef764a3de27dfdf1 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d586160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/0960e77e-2d9c-42d8-8ab3-81d22ea96970_isr_898695_45.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/0960e77e-2d9c-42d8-8ab3-81d22ea96970_isr_898695_45.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App characterizeImage, waiting on task 18 parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 19 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App calibrate, waiting on task 19 parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 20 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 53039eb29b019d6a4f1b93b16fccb6bb parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5863e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/dc06ec55-1012-4752-b547-ad469a8f2d70_isr_898798_4.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/dc06ec55-1012-4752-b547-ad469a8f2d70_isr_898798_4.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App characterizeImage, waiting on task 21 parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App calibrate, waiting on task 22 parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 23 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 056d08361cc28ccf8a264e446ee5a13b parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d587740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/500359ef-514c-45ab-92ed-76c8b73d22ce_isr_898695_62.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/500359ef-514c-45ab-92ed-76c8b73d22ce_isr_898695_62.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App characterizeImage, waiting on task 24 parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 25 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App calibrate, waiting on task 25 parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 26 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 36802b7279daacc60c2b4ebbff5d59f9 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d587420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/ddd01043-da94-4d1c-8f7c-d2485b9e7f24_isr_898696_18.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/ddd01043-da94-4d1c-8f7c-d2485b9e7f24_isr_898696_18.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App characterizeImage, waiting on task 27 parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 28 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App calibrate, waiting on task 28 parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 29 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash dc30609d9a18a54a656f8ba6745a50e3 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d587100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/a9b50873-8a87-4591-9ed1-337d960dcb8b_isr_898799_24.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/a9b50873-8a87-4591-9ed1-337d960dcb8b_isr_898799_24.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App characterizeImage, waiting on task 30 parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 31 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App calibrate, waiting on task 31 parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 32 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash e075d87a00ffef96973b32525900e95a parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d586de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/461dd6ca-6d6b-42c2-8f0d-c28bb64c0a24_isr_898800_36.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/461dd6ca-6d6b-42c2-8f0d-c28bb64c0a24_isr_898800_36.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App characterizeImage, waiting on task 33 parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 34 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App calibrate, waiting on task 34 parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 35 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 13782de9e644867cae7d9194119c26e6 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d586ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/fefaeac6-6037-479e-a8d6-eced1f42431c_isr_898696_42.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/fefaeac6-6037-479e-a8d6-eced1f42431c_isr_898696_42.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App characterizeImage, waiting on task 36 parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 37 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App calibrate, waiting on task 37 parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 38 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 305fbc72445a9596c7de7ab202d74407 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d586700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/6430d955-9eed-4af8-8334-a43a49db5885_isr_898798_61.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/6430d955-9eed-4af8-8334-a43a49db5885_isr_898798_61.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App characterizeImage, waiting on task 39 parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 40 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App calibrate, waiting on task 40 parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 41 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 5b96b2264812d9f68f9d7ad61405da51 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d587b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/333ca9f4-c8b9-4658-a3a9-a24eccb90d9b_isr_898694_18.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/333ca9f4-c8b9-4658-a3a9-a24eccb90d9b_isr_898694_18.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App characterizeImage, waiting on task 42 parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 43 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App calibrate, waiting on task 43 parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 44 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 80eef661c1459bfb7e7fc7b44e8bcb19 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d587e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/44e8c293-3410-482e-9664-8c0fa06896f1_isr_898799_49.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/44e8c293-3410-482e-9664-8c0fa06896f1_isr_898799_49.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App characterizeImage, waiting on task 45 parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 46 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App calibrate, waiting on task 46 parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 47 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 12c1d79c1d617d82470a49725dd8ec88 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/be1aa605-1176-49b2-be63-262138ac92b3_isr_898695_44.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/be1aa605-1176-49b2-be63-262138ac92b3_isr_898695_44.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App characterizeImage, waiting on task 48 parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 49 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App calibrate, waiting on task 49 parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 50 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash a1f8f38b32286e7fe94b4aad0e69d444 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/7eaa9d9c-2bb4-4d33-b123-7a7d3e0a86bd_isr_898696_41.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/7eaa9d9c-2bb4-4d33-b123-7a7d3e0a86bd_isr_898696_41.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App characterizeImage, waiting on task 51 parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 52 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App calibrate, waiting on task 52 parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 53 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash e91fed8770239c922b96c27059342754 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/01108099-1e3f-4d01-a3be-05b9b5237874_isr_898694_36.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/01108099-1e3f-4d01-a3be-05b9b5237874_isr_898694_36.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App characterizeImage, waiting on task 54 parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 55 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App calibrate, waiting on task 55 parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 56 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 3829ac15b68d38fcd5dd5ac5a4b5c206 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/0fad463c-cb87-42a8-9fc6-f88af96c633c_isr_898694_55.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/0fad463c-cb87-42a8-9fc6-f88af96c633c_isr_898694_55.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App characterizeImage, waiting on task 57 parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 58 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App calibrate, waiting on task 58 parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 59 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 5949ac70238af7e8cd424530585becbf parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/e010299a-81e3-43f1-975d-eb56c0a090c3_isr_898798_24.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/e010299a-81e3-43f1-975d-eb56c0a090c3_isr_898798_24.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App characterizeImage, waiting on task 60 parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 61 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App calibrate, waiting on task 61 parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 62 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 43f2d216a6bd968d5624a88a17b5ee58 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/8348b90c-4d73-4e1f-a027-92f6c242af6c_isr_898798_53.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/8348b90c-4d73-4e1f-a027-92f6c242af6c_isr_898798_53.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App characterizeImage, waiting on task 63 parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 64 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App calibrate, waiting on task 64 parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 65 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 6bff183d9f7ebe5a7fb4f138422a0633 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/ecffa768-876b-4cc4-88b8-e4c27427ca2e_isr_898800_7.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/ecffa768-876b-4cc4-88b8-e4c27427ca2e_isr_898800_7.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App characterizeImage, waiting on task 66 parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 67 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App calibrate, waiting on task 67 parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 68 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 14d995634fa0c397508d8002f34e01da parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/e54fe647-726d-4c05-8d12-dbf029c96829_isr_898694_50.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/e54fe647-726d-4c05-8d12-dbf029c96829_isr_898694_50.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App characterizeImage, waiting on task 69 parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 70 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App calibrate, waiting on task 70 parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 71 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 620182c3ff1b3a3ddff7f29cb38624f5 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/04ad5366-d984-4c84-9a67-fd8711aa8c75_isr_898799_46.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/04ad5366-d984-4c84-9a67-fd8711aa8c75_isr_898799_46.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App characterizeImage, waiting on task 72 parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 73 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App calibrate, waiting on task 73 parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 74 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash c9ceca399ac572a11d634d706af8f9c4 parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/0f9c9a5f-dc72-4720-be49-bac45f1e9f95_isr_898695_58.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/0f9c9a5f-dc72-4720-be49-bac45f1e9f95_isr_898695_58.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App characterizeImage, waiting on task 75 parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 76 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App calibrate, waiting on task 76 parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 77 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash d6659cb25abb9214fd34e2e03a099d67 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5aa0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/f665bb70-2d5e-48ef-a435-95f7ef5510e7_isr_898800_1.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/f665bb70-2d5e-48ef-a435-95f7ef5510e7_isr_898800_1.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App characterizeImage, waiting on task 78 parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 79 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App calibrate, waiting on task 79 parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 80 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 10be0233bafe76c4301491f657e4b859 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5aa3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/6c497fd7-306e-40e7-ab8b-026b8debd94d_isr_898798_52.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/6c497fd7-306e-40e7-ab8b-026b8debd94d_isr_898798_52.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App characterizeImage, waiting on task 81 parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 82 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App calibrate, waiting on task 82 parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 83 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 9db6bfdbf973c6932a7c077a7c9ff62f parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5aa700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/9ccabdfc-acff-490d-b558-37f0f851a813_isr_898800_58.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/9ccabdfc-acff-490d-b558-37f0f851a813_isr_898800_58.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App characterizeImage, waiting on task 84 parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 85 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App calibrate, waiting on task 85 parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 86 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 0a8810c0ffbdc053f327f7f72e5e54bb parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5aaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/6c65040a-3fa0-4ffe-9c3d-e1a20175109c_isr_898695_47.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/6c65040a-3fa0-4ffe-9c3d-e1a20175109c_isr_898695_47.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App characterizeImage, waiting on task 87 parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 88 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App calibrate, waiting on task 88 parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 89 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash d08df0628c88cb430fcc6a52ad09501c parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5aad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/b34b796e-f044-4fc2-b421-7b57177e8269_isr_898694_48.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/b34b796e-f044-4fc2-b421-7b57177e8269_isr_898694_48.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App characterizeImage, waiting on task 90 parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 91 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App calibrate, waiting on task 91 parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 92 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 97b95643e162322737e037fb0fdb0dcc parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ab060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/4a86629b-82a9-4f29-81cd-739e3717df4f_isr_898696_20.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/4a86629b-82a9-4f29-81cd-739e3717df4f_isr_898696_20.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App characterizeImage, waiting on task 93 parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 94 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App calibrate, waiting on task 94 parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 95 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash f4de89dbd03caf405be4c77d1237805c parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ab380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/e02d2029-da94-49b6-8568-40d1bf3182ce_isr_898799_36.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/e02d2029-da94-49b6-8568-40d1bf3182ce_isr_898799_36.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App characterizeImage, waiting on task 96 parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 97 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App calibrate, waiting on task 97 parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 98 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash a84828b781324d98abb246ce341ed363 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ab6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/2fa7de8b-3992-43a6-a3c5-c3345c9dc37a_isr_898798_46.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/2fa7de8b-3992-43a6-a3c5-c3345c9dc37a_isr_898798_46.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App characterizeImage, waiting on task 99 parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 100 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App calibrate, waiting on task 100 parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 101 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash d6d1e7f0cdb2ecc73d36566e035b7481 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ab9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/767ebc18-f779-4184-83b6-f6cb408f7c80_isr_898695_60.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/767ebc18-f779-4184-83b6-f6cb408f7c80_isr_898695_60.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App characterizeImage, waiting on task 102 parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App calibrate, waiting on task 103 parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 104 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash d594d037e1d9094d158d9a2e7149bce8 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5abce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/bc7fc3ee-4f74-4223-8ff7-ebdf425ed777_isr_898799_58.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/bc7fc3ee-4f74-4223-8ff7-ebdf425ed777_isr_898799_58.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App characterizeImage, waiting on task 105 parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 106 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App calibrate, waiting on task 106 parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 107 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 1d15a98ee78ff12c4865ab252c89a5fa parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cc040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/f9c57548-e641-4ebe-99ad-1987bdcba294_isr_898798_32.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/f9c57548-e641-4ebe-99ad-1987bdcba294_isr_898798_32.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App characterizeImage, waiting on task 108 parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App calibrate, waiting on task 109 parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 110 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 6a46b2ebcc7671c9a40687fa51c5a457 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cc360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/40fdb079-5cbf-4151-aa47-5aafaa5d162e_isr_898694_62.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/40fdb079-5cbf-4151-aa47-5aafaa5d162e_isr_898694_62.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App characterizeImage, waiting on task 111 parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 112 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App calibrate, waiting on task 112 parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 113 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 56aa47348f1732e3f987d423418b2fbc parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cc680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/90e7fd80-d2eb-4644-a250-95adb1e7be83_isr_898800_54.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/90e7fd80-d2eb-4644-a250-95adb1e7be83_isr_898800_54.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App characterizeImage, waiting on task 114 parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 115 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App calibrate, waiting on task 115 parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 116 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash e1a9d92196b42d6c3e5a5610513214d6 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/432efd01-601b-4868-bcbb-d47ceded1578_isr_898696_13.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/432efd01-601b-4868-bcbb-d47ceded1578_isr_898696_13.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App characterizeImage, waiting on task 117 parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 118 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App calibrate, waiting on task 118 parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 119 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 1b2a43b561b8ad77c048da6cc5f4fb15 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/bead010d-ea01-404a-8fee-dceaa12a264f_isr_898695_55.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/bead010d-ea01-404a-8fee-dceaa12a264f_isr_898695_55.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App characterizeImage, waiting on task 120 parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 121 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App calibrate, waiting on task 121 parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 122 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 3ea853824217f1052b79b6e8cc061f35 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ccfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/e33aa75b-33e9-4463-9a9a-339c43e64d89_isr_898694_53.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/e33aa75b-33e9-4463-9a9a-339c43e64d89_isr_898694_53.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App characterizeImage, waiting on task 123 parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 124 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App calibrate, waiting on task 124 parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 125 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 3f1b0148dd7fa71b1554d52f5f94e3d7 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cd300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/dc72a31a-1e00-4b35-963a-bb1605741efe_isr_898695_35.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/dc72a31a-1e00-4b35-963a-bb1605741efe_isr_898695_35.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App characterizeImage, waiting on task 126 parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 127 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App calibrate, waiting on task 127 parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 128 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 19eb34e720d5871322820c9c4cc80e40 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/bd2d4af7-7aa3-449d-8c38-a0e0a281dbc8_isr_898694_57.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/bd2d4af7-7aa3-449d-8c38-a0e0a281dbc8_isr_898694_57.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App characterizeImage, waiting on task 129 parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 130 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App calibrate, waiting on task 130 parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 131 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 284efad1011be4f11648d9ece29f15a1 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/8b7363ac-e048-4509-906d-526766328889_isr_898695_25.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/8b7363ac-e048-4509-906d-526766328889_isr_898695_25.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App characterizeImage, waiting on task 132 parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 133 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App calibrate, waiting on task 133 parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 134 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash f3981bcd20ec66c6586411d6d25148e7 parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cdc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/f0f96198-001a-4edf-bdf2-acd5b50859c1_isr_898694_47.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/f0f96198-001a-4edf-bdf2-acd5b50859c1_isr_898694_47.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App characterizeImage, waiting on task 135 parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 136 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App calibrate, waiting on task 136 parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 137 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash b783d7f2b854d35f93c76c773f250625 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cdf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/50415f15-e2c6-4ee7-9948-797c2d2729ae_isr_898800_40.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/50415f15-e2c6-4ee7-9948-797c2d2729ae_isr_898800_40.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App characterizeImage, waiting on task 138 parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 139 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App calibrate, waiting on task 139 parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 140 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash f24e0266c101ab5683cf54e2e2d9697b parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ce2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/0f656dc7-4c66-4a89-91a4-4bc06e21ecb1_isr_898799_10.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/0f656dc7-4c66-4a89-91a4-4bc06e21ecb1_isr_898799_10.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App characterizeImage, waiting on task 141 parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 142 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App calibrate, waiting on task 142 parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 143 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 0f8b588649491f006e560cbdb7a37ec6 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ce5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/79a4f861-5b37-4c6d-9c36-9ee5e20a4d79_isr_898695_17.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/79a4f861-5b37-4c6d-9c36-9ee5e20a4d79_isr_898695_17.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App characterizeImage, waiting on task 144 parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 145 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App calibrate, waiting on task 145 parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 146 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash fcda30ba244e0c15b2d47fb384a788fd parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ce8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/33902bee-b3ba-424a-9f21-aa6654459ea5_isr_898694_44.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/33902bee-b3ba-424a-9f21-aa6654459ea5_isr_898694_44.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App characterizeImage, waiting on task 147 parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 148 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App calibrate, waiting on task 148 parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 149 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 162cfd913f0cef137f2c66ead3534f64 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/a9a4331d-d21b-419f-92dc-99761cd333e1_isr_898800_15.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/a9a4331d-d21b-419f-92dc-99761cd333e1_isr_898800_15.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App characterizeImage, waiting on task 150 parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 151 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App calibrate, waiting on task 151 parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 152 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 7fe2e4361bd18139f7a6ebaaea119727 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/0e9ef9f9-1102-470b-b2f3-f698b08128a6_isr_898694_46.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/0e9ef9f9-1102-470b-b2f3-f698b08128a6_isr_898694_46.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App characterizeImage, waiting on task 153 parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 154 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App calibrate, waiting on task 154 parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 155 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash fd3929e86eacdcf283d5c9fd9732cd0a parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cf240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/45355476-83d9-4d27-9ffd-1b869dd00fcd_isr_898695_4.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/45355476-83d9-4d27-9ffd-1b869dd00fcd_isr_898695_4.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App characterizeImage, waiting on task 156 parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 157 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App calibrate, waiting on task 157 parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 158 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 59ded46f6d7d87880d82152a958f7f7a parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cf560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/f07d52b1-141d-4f71-afc2-206e2125a55d_isr_898695_51.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/f07d52b1-141d-4f71-afc2-206e2125a55d_isr_898695_51.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App characterizeImage, waiting on task 159 parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 160 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App calibrate, waiting on task 160 parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 161 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash ba4293e5c6c6a56f78908079daa78531 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cf880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/eb9405da-da31-4803-8355-19bff0f2a8d1_isr_898800_27.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/eb9405da-da31-4803-8355-19bff0f2a8d1_isr_898800_27.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App characterizeImage, waiting on task 162 parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 163 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App calibrate, waiting on task 163 parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 164 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash d219ec03d9fb66d60af1a3f59029cd82 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cfba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/acd147a1-6f80-4dd2-81eb-b6c6b1bbda6a_isr_898696_52.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/acd147a1-6f80-4dd2-81eb-b6c6b1bbda6a_isr_898696_52.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App characterizeImage, waiting on task 165 parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 166 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App calibrate, waiting on task 166 parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 167 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 02db1b7797f84d3f3d35239769946ee0 parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cfec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/c7b5407e-9c09-43f3-ba55-0d8acb6fd95b_isr_898800_31.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/c7b5407e-9c09-43f3-ba55-0d8acb6fd95b_isr_898800_31.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App characterizeImage, waiting on task 168 parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 169 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App calibrate, waiting on task 169 parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 170 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash eb37d29d10305314a55de599e8139c92 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ec220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/2e9c7c9c-80df-4eba-af8c-31df9385fb4b_isr_898694_29.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/2e9c7c9c-80df-4eba-af8c-31df9385fb4b_isr_898694_29.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App characterizeImage, waiting on task 171 parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 172 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App calibrate, waiting on task 172 parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 173 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 4788002c0e167b07e9d9c83c660336d2 parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ec540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/217639e6-bc7f-46e2-9ebf-57221adbd217_isr_898798_33.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/217639e6-bc7f-46e2-9ebf-57221adbd217_isr_898798_33.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App characterizeImage, waiting on task 174 parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 175 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App calibrate, waiting on task 175 parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 176 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 0284393415845ee508d706696c49eeb2 parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ec860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/2ac7b472-aa0d-47cc-b73b-d8987e10bad4_isr_898800_24.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/2ac7b472-aa0d-47cc-b73b-d8987e10bad4_isr_898800_24.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App characterizeImage, waiting on task 177 parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 178 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App calibrate, waiting on task 178 parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 179 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash c083dec7f3cf5147faa0c2daf945fb0e parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ecb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/802287e0-c245-4765-9cb6-abf69986e5fe_isr_898800_41.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/802287e0-c245-4765-9cb6-abf69986e5fe_isr_898800_41.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App characterizeImage, waiting on task 180 parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 181 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App calibrate, waiting on task 181 parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 182 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 3fabff2bb6db9c5d1cc2d4618e33fbf4 parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ecea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/2bf0ec7c-5e83-4cbe-a37e-ee51caf3034d_isr_898695_48.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/2bf0ec7c-5e83-4cbe-a37e-ee51caf3034d_isr_898695_48.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App characterizeImage, waiting on task 183 parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 184 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App calibrate, waiting on task 184 parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 185 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 23a8e417daa4de8d4705a465581ff178 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ed1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/6c069d3f-4059-441f-ba8d-f2e6d654e038_isr_898696_5.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/6c069d3f-4059-441f-ba8d-f2e6d654e038_isr_898696_5.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App characterizeImage, waiting on task 186 parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 187 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App calibrate, waiting on task 187 parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 188 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash a0fb715dc109fd626bb16399e56441e0 parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ed4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/5f3a6e0e-61c7-4413-857e-0603b888709e_isr_898696_55.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/5f3a6e0e-61c7-4413-857e-0603b888709e_isr_898696_55.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App characterizeImage, waiting on task 189 parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 190 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App calibrate, waiting on task 190 parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 191 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash d4c95bd1c45381ae26c712feacb099ba parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ed800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/9ce3da53-bfcf-458b-a0da-1bab2002ccd2_isr_898799_37.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/9ce3da53-bfcf-458b-a0da-1bab2002ccd2_isr_898799_37.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App characterizeImage, waiting on task 192 parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 193 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App calibrate, waiting on task 193 parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 194 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 9ee8e5116410f9555d6286ed9a3f8410 parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5edb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/93d98c56-b6b0-46d1-818e-8a890f91f2a6_isr_898694_31.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/93d98c56-b6b0-46d1-818e-8a890f91f2a6_isr_898694_31.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App characterizeImage, waiting on task 195 parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 196 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App calibrate, waiting on task 196 parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 197 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 27b4c13eeafc24f1d07b73b579ce6795 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ede40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/415d2eed-39c3-4c69-afcb-d948a123e5fb_isr_898695_18.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/415d2eed-39c3-4c69-afcb-d948a123e5fb_isr_898695_18.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App characterizeImage, waiting on task 198 parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 199 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App calibrate, waiting on task 199 parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 200 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 1fcd3bb5bca48e2415fe3324f86fb0fc parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ee160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/38a5576c-b5cc-4dc0-a983-b416b6697b46_isr_898694_5.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/38a5576c-b5cc-4dc0-a983-b416b6697b46_isr_898694_5.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App characterizeImage, waiting on task 201 parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 202 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App calibrate, waiting on task 202 parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 203 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 2cf18bf6d38b96e649ecfa8a4ada1929 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ee480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/cbc3ebad-eada-4d1a-92d3-99de91fda053_isr_898695_41.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/cbc3ebad-eada-4d1a-92d3-99de91fda053_isr_898695_41.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App characterizeImage, waiting on task 204 parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 205 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App calibrate, waiting on task 205 parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 206 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 3c3f1ec22574722b6bb3e0110e4ca34c parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ee7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/ff3c5cc2-984a-4975-8d75-fd05f0eec56f_isr_898799_8.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/ff3c5cc2-984a-4975-8d75-fd05f0eec56f_isr_898799_8.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App characterizeImage, waiting on task 207 parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 208 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App calibrate, waiting on task 208 parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 209 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 567b582a4ce2911b310a322fc0a5a558 parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5eeac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/d501d583-8718-4e20-8f9a-be5d228f6823_isr_898798_13.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/d501d583-8718-4e20-8f9a-be5d228f6823_isr_898798_13.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App characterizeImage, waiting on task 210 parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 211 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App calibrate, waiting on task 211 parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 212 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 3e6fdd9211acd29caa18a0486aac280e parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5eede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/5046e09e-0387-4a52-9fcb-915aefdceeb0_isr_898694_60.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/5046e09e-0387-4a52-9fcb-915aefdceeb0_isr_898694_60.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App characterizeImage, waiting on task 213 parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 214 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App calibrate, waiting on task 214 parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 215 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash e89498a2ab2404314fa6bf33240e3536 parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ef100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/21ddfde0-e67e-4bed-9a0e-11576762d1eb_isr_898799_23.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/21ddfde0-e67e-4bed-9a0e-11576762d1eb_isr_898799_23.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App characterizeImage, waiting on task 216 parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 217 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App calibrate, waiting on task 217 parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 218 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 1a589b9eaa0b06841f7efa931efc78db parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ef420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/aae9d5a4-7741-4c09-ac49-7ed2200020ed_isr_898799_4.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/aae9d5a4-7741-4c09-ac49-7ed2200020ed_isr_898799_4.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App characterizeImage, waiting on task 219 parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 220 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App calibrate, waiting on task 220 parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 221 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash 43628b794b317243b590790212133035 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ef740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/f0ceac77-3fab-437a-9e27-0831f6bf4d62_isr_898798_10.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/f0ceac77-3fab-437a-9e27-0831f6bf4d62_isr_898798_10.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App characterizeImage, waiting on task 222 parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 223 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App calibrate, waiting on task 223 parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 224 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 62ba6aac74a7b005e340bddbffbec664 parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5efa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/ba3cc9cb-6e90-414f-b91b-39a50a18beca_isr_898800_12.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/ba3cc9cb-6e90-414f-b91b-39a50a18beca_isr_898800_12.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App characterizeImage, waiting on task 225 parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 226 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App calibrate, waiting on task 226 parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 227 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash 9a9a629d36b8ffd67e5b4e7e38f6b048 parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5efd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/fbfffb24-4a12-4f0a-a34a-c7d916f930e5_isr_898798_41.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/fbfffb24-4a12-4f0a-a34a-c7d916f930e5_isr_898798_41.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App characterizeImage, waiting on task 228 parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 229 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App calibrate, waiting on task 229 parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 230 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash bf0a424849e63e43acd02facbb06d9f3 parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6080e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/c997ef57-3a46-4752-8a97-58a4cf7e4ec4_isr_898696_21.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/c997ef57-3a46-4752-8a97-58a4cf7e4ec4_isr_898696_21.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App characterizeImage, waiting on task 231 parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 232 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App calibrate, waiting on task 232 parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 233 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash e3e3846130926c80f5c12b69ce5f61a4 parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d608400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/94ca82bd-1921-4d02-a2cc-e610008c0956_isr_898799_42.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/94ca82bd-1921-4d02-a2cc-e610008c0956_isr_898799_42.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App characterizeImage, waiting on task 234 parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 235 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App calibrate, waiting on task 235 parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 236 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 46626ecfad57315e6464e263d48cc79a parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d608720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/747b8428-1613-45b7-8eff-d5dc5f5fabba_isr_898695_39.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/747b8428-1613-45b7-8eff-d5dc5f5fabba_isr_898695_39.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App characterizeImage, waiting on task 237 parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 238 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App calibrate, waiting on task 238 parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 239 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash beafc005887a49b15d216862ee6e2dd1 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d608a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/70184483-6012-478f-b865-a7d4f72924fa_isr_898800_18.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/70184483-6012-478f-b865-a7d4f72924fa_isr_898800_18.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App characterizeImage, waiting on task 240 parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 241 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App calibrate, waiting on task 241 parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 242 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash 7247ecdac63873b3eaf1e1eeee5e0e15 parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d608d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/8f846a96-bfdc-41d7-a64a-1c7516546aba_isr_898799_2.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/8f846a96-bfdc-41d7-a64a-1c7516546aba_isr_898799_2.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App characterizeImage, waiting on task 243 parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 244 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App calibrate, waiting on task 244 parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 245 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 589eb1fcce48cc93de03f51ff0cdf9d5 parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d609080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/bcbf8fd9-60cb-42df-ae7a-e79a02485a76_isr_898694_41.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/bcbf8fd9-60cb-42df-ae7a-e79a02485a76_isr_898694_41.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App characterizeImage, waiting on task 246 parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 247 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App calibrate, waiting on task 247 parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 248 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 5a7881f538950abfe58032f394f67f7d parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6093a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/099eaac4-c208-4fe8-aafd-92229ac3b25d_isr_898694_42.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/099eaac4-c208-4fe8-aafd-92229ac3b25d_isr_898694_42.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App characterizeImage, waiting on task 249 parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 250 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App calibrate, waiting on task 250 parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 251 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash 6ee4de340a41398f9435513edca1026e parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6096c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/cbe58183-d846-43c0-a966-3d898f74fb6c_isr_898799_32.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/cbe58183-d846-43c0-a966-3d898f74fb6c_isr_898799_32.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App characterizeImage, waiting on task 252 parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 253 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App calibrate, waiting on task 253 parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 254 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash 504de62121f961743d4b452a2960cd58 parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6099e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/3cbcbf24-c16c-4e57-a67e-3485d4e316ea_isr_898798_23.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/3cbcbf24-c16c-4e57-a67e-3485d4e316ea_isr_898798_23.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App characterizeImage, waiting on task 255 parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 256 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App calibrate, waiting on task 256 parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 257 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash 5d7d72916f44a3dced4f34665e671e2e parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d609d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/bcf59e8b-04ca-4e6f-a2ca-0f66228b9e0b_isr_898695_29.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/bcf59e8b-04ca-4e6f-a2ca-0f66228b9e0b_isr_898695_29.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App characterizeImage, waiting on task 258 parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 259 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App calibrate, waiting on task 259 parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 260 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash 1fdf5bde54fb6ee92b8c1dc0b634f908 parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/ac348d27-485c-4315-a55c-12ddfbd688e6_isr_898798_54.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/ac348d27-485c-4315-a55c-12ddfbd688e6_isr_898798_54.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App characterizeImage, waiting on task 261 parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 262 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App calibrate, waiting on task 262 parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 263 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash ba09f6f78575739e856c05aae3f20a6f parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/1892b3a4-7ab4-4bff-8c71-d2b0a2d159c0_isr_898696_35.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/1892b3a4-7ab4-4bff-8c71-d2b0a2d159c0_isr_898696_35.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App characterizeImage, waiting on task 264 parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 265 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App calibrate, waiting on task 265 parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 266 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash ae9e4a25770551756166d55c1a6952b3 parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/36d7c93d-b7a4-4d33-95cf-ac90330d481f_isr_898800_46.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/36d7c93d-b7a4-4d33-95cf-ac90330d481f_isr_898800_46.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App characterizeImage, waiting on task 267 parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 268 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App calibrate, waiting on task 268 parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 269 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash ff84ceae97ac18903ec025452f3494eb parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/cf5d0c2b-a256-4950-9626-9e98bee6cf14_isr_898799_52.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/cf5d0c2b-a256-4950-9626-9e98bee6cf14_isr_898799_52.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App characterizeImage, waiting on task 270 parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 271 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App calibrate, waiting on task 271 parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 272 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash bb5208ea1b6642b3cec92d48e0a60743 parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/8b79a977-4446-459a-be22-54db5a785cd8_isr_898800_60.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/8b79a977-4446-459a-be22-54db5a785cd8_isr_898800_60.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App characterizeImage, waiting on task 273 parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 274 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App calibrate, waiting on task 274 parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 275 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash 9f86f5799f143cd68475e4a8cfe6bdf2 parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/9fb7bb20-5c8c-4d19-b003-9b339baf98df_isr_898799_18.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/9fb7bb20-5c8c-4d19-b003-9b339baf98df_isr_898799_18.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App characterizeImage, waiting on task 276 parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 277 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App calibrate, waiting on task 277 parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 278 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 5a6e709d82ae0ec265bd7512fc2ae308 parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/31993d28-3800-4e3f-9fb6-63f3d4d617cc_isr_898694_10.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/31993d28-3800-4e3f-9fb6-63f3d4d617cc_isr_898694_10.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App characterizeImage, waiting on task 279 parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 280 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App calibrate, waiting on task 280 parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 281 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash b1ae022506a5912de8a2748c150d97c6 parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/bd93033b-2e96-4ab4-9215-4885aeb9bc04_isr_898799_47.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/bd93033b-2e96-4ab4-9215-4885aeb9bc04_isr_898799_47.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App characterizeImage, waiting on task 282 parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 283 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App calibrate, waiting on task 283 parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 284 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash bb8864a6aad2bf68cad2dc460f029d82 parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/1c599cda-09f8-4f73-9ac0-d72b2d001e36_isr_898800_57.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/1c599cda-09f8-4f73-9ac0-d72b2d001e36_isr_898800_57.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App characterizeImage, waiting on task 285 parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 286 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App calibrate, waiting on task 286 parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 287 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash 0fb33769887e703097537b3761493a7f parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/d9e45d5c-05bb-40f5-a869-37e8a52141f6_isr_898694_3.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/d9e45d5c-05bb-40f5-a869-37e8a52141f6_isr_898694_3.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App characterizeImage, waiting on task 288 parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 289 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App calibrate, waiting on task 289 parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 290 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash a921de290bf37e5261dbed1e1ce9e1b8 parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/813ed576-93b8-4b5f-8190-d265fc2dc6c0_isr_898694_58.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/813ed576-93b8-4b5f-8190-d265fc2dc6c0_isr_898694_58.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App characterizeImage, waiting on task 291 parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 292 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App calibrate, waiting on task 292 parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 293 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash cd2779e5f74cf2c980aaa93cfd3e3093 parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/52cbf908-57ce-49d2-bbdf-01664b9e7b3e_isr_898694_54.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/52cbf908-57ce-49d2-bbdf-01664b9e7b3e_isr_898694_54.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App characterizeImage, waiting on task 294 parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 295 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App calibrate, waiting on task 295 parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 296 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 1d3069fa279c4cc447967e78fbbdde8d parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/a5d0bbfb-87ac-4b11-9699-1bdcb6e79933_isr_898799_22.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/a5d0bbfb-87ac-4b11-9699-1bdcb6e79933_isr_898799_22.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App characterizeImage, waiting on task 297 parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 298 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App calibrate, waiting on task 298 parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 299 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash cfe3c0296f98489ec2f77171adbc2ade parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/f66c1442-4eb1-4ff7-8b3f-e1ad483199f3_isr_898696_59.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/f66c1442-4eb1-4ff7-8b3f-e1ad483199f3_isr_898696_59.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App characterizeImage, waiting on task 300 parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 301 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App calibrate, waiting on task 301 parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 302 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash ebdba6361011041646ad1514594ed46b parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/2a58b74a-c610-4b10-9ed9-88aba1d6115d_isr_898694_14.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/2a58b74a-c610-4b10-9ed9-88aba1d6115d_isr_898694_14.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App characterizeImage, waiting on task 303 parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 304 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App calibrate, waiting on task 304 parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 305 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash bb8846e01eaeac367644b3c3e0f42ed5 parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/d4763365-540c-4c1d-b593-d2112f23a8d6_isr_898696_26.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/d4763365-540c-4c1d-b593-d2112f23a8d6_isr_898696_26.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App characterizeImage, waiting on task 306 parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 307 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App calibrate, waiting on task 307 parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 308 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 479f3a419104edc74176f6c58a829e0c parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/1f11f0b3-4243-45f9-a7f2-50a948edba27_isr_898799_25.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/1f11f0b3-4243-45f9-a7f2-50a948edba27_isr_898799_25.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App characterizeImage, waiting on task 309 parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 310 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App calibrate, waiting on task 310 parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 311 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash e048e3218ad058a92a9e66d35302304c parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/9165cfcd-7e59-44bb-8cb5-637d987c53fd_isr_898799_45.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/9165cfcd-7e59-44bb-8cb5-637d987c53fd_isr_898799_45.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App characterizeImage, waiting on task 312 parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 313 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App calibrate, waiting on task 313 parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 314 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash 0934989a531a37b186e08a957b8871bf parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/5ef55cb0-dc80-47aa-a1a6-7f34a2a60495_isr_898694_1.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/5ef55cb0-dc80-47aa-a1a6-7f34a2a60495_isr_898694_1.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App characterizeImage, waiting on task 315 parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 316 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App calibrate, waiting on task 316 parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 317 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash c757da3a19e7c564c8f561a9af0ced26 parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/b58144c6-66dd-4ff9-bfcf-d4e5c07afa0a_isr_898798_17.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/b58144c6-66dd-4ff9-bfcf-d4e5c07afa0a_isr_898798_17.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App characterizeImage, waiting on task 318 parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 319 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App calibrate, waiting on task 319 parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 320 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 419e2c93b19d43c2c18aef5d957f5eb2 parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/c0a0c7b1-1ccf-405d-9d32-7b92b534c6f0_isr_898695_19.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/c0a0c7b1-1ccf-405d-9d32-7b92b534c6f0_isr_898695_19.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App characterizeImage, waiting on task 321 parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 322 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App calibrate, waiting on task 322 parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 323 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash 3c8d40c29dba8c1f4e054cdb955bdbf5 parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/6b618ae3-431b-4d5d-a65b-1870efd6b707_isr_898695_26.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/6b618ae3-431b-4d5d-a65b-1870efd6b707_isr_898695_26.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App characterizeImage, waiting on task 324 parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 325 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App calibrate, waiting on task 325 parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 326 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash 8bdb190fd389956c27a7763eea64ee55 parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/76a793c9-c574-4fc2-92ff-2d09bca463ec_isr_898694_25.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/76a793c9-c574-4fc2-92ff-2d09bca463ec_isr_898694_25.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App characterizeImage, waiting on task 327 parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 328 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App calibrate, waiting on task 328 parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 329 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 8cadc4829f0a8ca5ddb0f968ad63a4d0 parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/a42f0707-6139-4f6a-8c0b-fefd8e178ea6_isr_898696_12.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/a42f0707-6139-4f6a-8c0b-fefd8e178ea6_isr_898696_12.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App characterizeImage, waiting on task 330 parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 331 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App calibrate, waiting on task 331 parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 332 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash 8075a42f908e26fc924f721726568f54 parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/410e4da0-7f12-49bd-8b23-9633dda9e433_isr_898799_5.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/410e4da0-7f12-49bd-8b23-9633dda9e433_isr_898799_5.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App characterizeImage, waiting on task 333 parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 334 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App calibrate, waiting on task 334 parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 335 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash b07cca265dfa98f724abcbe5f6ce6521 parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/99526a73-6d1e-4121-a6da-96aea938d8ff_isr_898696_28.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/99526a73-6d1e-4121-a6da-96aea938d8ff_isr_898696_28.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App characterizeImage, waiting on task 336 parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 337 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App calibrate, waiting on task 337 parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 338 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash df74e0df9cf77f825e762809937592cc parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/68f93f20-aa37-4764-9c4e-0592282a4efa_isr_898694_15.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/68f93f20-aa37-4764-9c4e-0592282a4efa_isr_898694_15.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App characterizeImage, waiting on task 339 parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 340 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App calibrate, waiting on task 340 parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 341 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash b3ac1e4e08c2668a003fdd36a4ea1a51 parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/d8f5536f-1a7a-43cb-b646-d6115cffa3ae_isr_898695_54.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/d8f5536f-1a7a-43cb-b646-d6115cffa3ae_isr_898695_54.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App characterizeImage, waiting on task 342 parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 343 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App calibrate, waiting on task 343 parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 344 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash 276e40f201ff15fb4e6724fbc3d5568e parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/d18a9ac7-9b0d-4dda-8e82-3915f972633c_isr_898799_27.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/d18a9ac7-9b0d-4dda-8e82-3915f972633c_isr_898799_27.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App characterizeImage, waiting on task 345 parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 346 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App calibrate, waiting on task 346 parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 347 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash c28243152475bd4ed482dc825c9c6649 parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/4ab4ad42-0553-4adb-af27-fe36f2df2355_isr_898799_15.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/4ab4ad42-0553-4adb-af27-fe36f2df2355_isr_898799_15.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App characterizeImage, waiting on task 348 parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 349 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App calibrate, waiting on task 349 parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 350 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash e88e42d46fe0dc34c400d42260c3070a parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/72f31e36-6f13-477d-b3f5-b388a83f4e3e_isr_898696_2.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/72f31e36-6f13-477d-b3f5-b388a83f4e3e_isr_898696_2.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App characterizeImage, waiting on task 351 parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 352 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App calibrate, waiting on task 352 parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 353 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash 85aad82836761243bba754bff6dc2907 parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d660180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/6fc7ee91-245c-4583-8cc2-a4670a35ce15_isr_898695_12.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/6fc7ee91-245c-4583-8cc2-a4670a35ce15_isr_898695_12.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App characterizeImage, waiting on task 354 parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 355 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App calibrate, waiting on task 355 parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 356 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash bcacf20f1e0b71c150f1b558924b8a21 parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6604a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/8a4774b5-bb47-4531-be36-312ee00da89f_isr_898694_56.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/8a4774b5-bb47-4531-be36-312ee00da89f_isr_898694_56.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App characterizeImage, waiting on task 357 parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 358 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App calibrate, waiting on task 358 parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 359 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash 13a8336aba97555881b3bb5f6e1a7b1b parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6607c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/b3bb910a-acea-4dca-aa77-31f5eb7f7bce_isr_898695_27.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/b3bb910a-acea-4dca-aa77-31f5eb7f7bce_isr_898695_27.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App characterizeImage, waiting on task 360 parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 361 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App calibrate, waiting on task 361 parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 362 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash 4d8fad8aba8f43075bb8798e8eba9dae parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d660ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/07b4d63b-d3c1-4275-959d-ed59e46e2eb5_isr_898695_49.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/07b4d63b-d3c1-4275-959d-ed59e46e2eb5_isr_898695_49.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App characterizeImage, waiting on task 363 parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 364 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App calibrate, waiting on task 364 parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 365 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash 25daa97b3a9e5535b9116128119d8a86 parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d660e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/57cd5305-9d70-4b75-afe5-05e223cd36a4_isr_898695_40.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/57cd5305-9d70-4b75-afe5-05e223cd36a4_isr_898695_40.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App characterizeImage, waiting on task 366 parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 367 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App calibrate, waiting on task 367 parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 368 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash edee5859b706c3a347e460adf71a5b51 parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d661120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/f81c8bab-1c17-4627-908f-994ca20e90fa_isr_898696_38.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/f81c8bab-1c17-4627-908f-994ca20e90fa_isr_898696_38.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App characterizeImage, waiting on task 369 parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 370 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App calibrate, waiting on task 370 parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 371 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 372 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 372 has memoization hash 5b5b4157db5f714fcadf66d4420c5e27 parsl.dataflow.memoization INFO: Task 372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d661440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/974283d0-2718-4039-b583-a4af09f46084_isr_898696_43.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/974283d0-2718-4039-b583-a4af09f46084_isr_898696_43.stderr parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 373 submitted for App characterizeImage, waiting on task 372 parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 373 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 374 submitted for App calibrate, waiting on task 373 parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 374 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 375 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 375 has memoization hash 3146aea599e59e9b9f772bde42bf2834 parsl.dataflow.memoization INFO: Task 375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d661760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/2d2f834d-e885-4b61-b86c-5d951d0ddf60_isr_898694_40.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/2d2f834d-e885-4b61-b86c-5d951d0ddf60_isr_898694_40.stderr parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 376 submitted for App characterizeImage, waiting on task 375 parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 376 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 377 submitted for App calibrate, waiting on task 376 parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 377 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 378 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 378 has memoization hash 4ae000729cc0b8a58607e1caf114bf38 parsl.dataflow.memoization INFO: Task 378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d661a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/3bdd9e22-f8b8-4245-9207-9a750c42f83e_isr_898798_38.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/3bdd9e22-f8b8-4245-9207-9a750c42f83e_isr_898798_38.stderr parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 379 submitted for App characterizeImage, waiting on task 378 parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 379 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 380 submitted for App calibrate, waiting on task 379 parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 380 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 381 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 381 has memoization hash d5f7dd8b87554d8a96252de4a6aeae9d parsl.dataflow.memoization INFO: Task 381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d661da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/4c59969a-cf44-4d2e-bf58-4e12eb4910fa_isr_898800_59.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/4c59969a-cf44-4d2e-bf58-4e12eb4910fa_isr_898800_59.stderr parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 382 submitted for App characterizeImage, waiting on task 381 parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 382 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 383 submitted for App calibrate, waiting on task 382 parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 383 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 384 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 384 has memoization hash 24ce2292ee99b1d8583ebcfc5246b1d9 parsl.dataflow.memoization INFO: Task 384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6620c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/39f80a79-d290-4793-9dac-42c3c68cfde0_isr_898800_26.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/39f80a79-d290-4793-9dac-42c3c68cfde0_isr_898800_26.stderr parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 385 submitted for App characterizeImage, waiting on task 384 parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 385 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 386 submitted for App calibrate, waiting on task 385 parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 386 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 387 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 387 has memoization hash e856ddf266f963a108c733d17ef39952 parsl.dataflow.memoization INFO: Task 387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6623e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/5e6cf434-aed2-485f-904f-e78763034f7c_isr_898694_38.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/5e6cf434-aed2-485f-904f-e78763034f7c_isr_898694_38.stderr parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 388 submitted for App characterizeImage, waiting on task 387 parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 388 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 389 submitted for App calibrate, waiting on task 388 parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 389 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 390 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 390 has memoization hash 4ff285a57d2e80457f29109db302582b parsl.dataflow.memoization INFO: Task 390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d662700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/c5d69f9a-890a-4e9d-a53d-7c6ad6bfcd72_isr_898695_2.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/c5d69f9a-890a-4e9d-a53d-7c6ad6bfcd72_isr_898695_2.stderr parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 391 submitted for App characterizeImage, waiting on task 390 parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 391 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 392 submitted for App calibrate, waiting on task 391 parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 392 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 393 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 393 has memoization hash 76bfee96c5f15566b72ecba9264f0b3e parsl.dataflow.memoization INFO: Task 393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d662a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/7b6e8027-6a97-4b26-9939-7505471b2947_isr_898799_6.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/7b6e8027-6a97-4b26-9939-7505471b2947_isr_898799_6.stderr parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 394 submitted for App characterizeImage, waiting on task 393 parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 394 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 395 submitted for App calibrate, waiting on task 394 parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 395 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 396 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 396 has memoization hash 481d6e7ff72b6780a438b755d4a10d98 parsl.dataflow.memoization INFO: Task 396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d662d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/b05ec6ba-2e4e-4fec-b5b3-892e79708b11_isr_898694_7.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/b05ec6ba-2e4e-4fec-b5b3-892e79708b11_isr_898694_7.stderr parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 397 submitted for App characterizeImage, waiting on task 396 parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 397 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 398 submitted for App calibrate, waiting on task 397 parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 398 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 399 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 399 has memoization hash bb80225445e8c58479117fd75703d4e7 parsl.dataflow.memoization INFO: Task 399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d663060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/9ddcf504-0d9f-40c6-99b0-a322f43a1c47_isr_898694_22.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/9ddcf504-0d9f-40c6-99b0-a322f43a1c47_isr_898694_22.stderr parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 400 submitted for App characterizeImage, waiting on task 399 parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 400 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 401 submitted for App calibrate, waiting on task 400 parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 401 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 402 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] 2024-03-13 09:45:09 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 402 has memoization hash 290c2601654747cc11c06254ae98f96d parsl.dataflow.memoization INFO: Task 402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d663380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/73527873-438d-4fcc-aaff-2d2c5b18bc23_isr_898800_62.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/73527873-438d-4fcc-aaff-2d2c5b18bc23_isr_898800_62.stderr parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 403 submitted for App characterizeImage, waiting on task 402 parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 403 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 404 submitted for App calibrate, waiting on task 403 parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 404 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 405 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 405 has memoization hash 4904bc3bede6e31c68934dae0d471972 parsl.dataflow.memoization INFO: Task 405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6636a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/2e45e633-ff91-49a1-81a4-949b01651591_isr_898800_25.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/2e45e633-ff91-49a1-81a4-949b01651591_isr_898800_25.stderr parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 406 submitted for App characterizeImage, waiting on task 405 parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 406 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 407 submitted for App calibrate, waiting on task 406 parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 407 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 408 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 408 has memoization hash c101dbc58d13590a319df800a5e6dcea parsl.dataflow.memoization INFO: Task 408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d663c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/44044ec4-d51f-4ce6-8b2f-9530d082ab69_isr_898799_7.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/44044ec4-d51f-4ce6-8b2f-9530d082ab69_isr_898799_7.stderr parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 409 submitted for App characterizeImage, waiting on task 408 parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 409 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 410 submitted for App calibrate, waiting on task 409 parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 410 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 411 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 411 has memoization hash f5707fe8966b4317d670ad4fa7295aa3 parsl.dataflow.memoization INFO: Task 411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d663f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/69f402cc-202f-4315-ab84-2970966a51ea_isr_898694_6.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/69f402cc-202f-4315-ab84-2970966a51ea_isr_898694_6.stderr parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 412 submitted for App characterizeImage, waiting on task 411 parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 412 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 413 submitted for App calibrate, waiting on task 412 parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 413 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 414 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 414 has memoization hash 7ec9d46e88c25a2cfe972869ab644852 parsl.dataflow.memoization INFO: Task 414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c480220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/dc3d9518-93a7-4d78-b482-1c256c0526b6_isr_898695_46.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/dc3d9518-93a7-4d78-b482-1c256c0526b6_isr_898695_46.stderr parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 415 submitted for App characterizeImage, waiting on task 414 parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 415 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 416 submitted for App calibrate, waiting on task 415 parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 416 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 417 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 417 has memoization hash ff4da2ae0b3c5638c2006755cd107cc4 parsl.dataflow.memoization INFO: Task 417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4802c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/c349fcb6-62e4-4242-bc76-7863a2894075_isr_898800_35.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/c349fcb6-62e4-4242-bc76-7863a2894075_isr_898800_35.stderr parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 418 submitted for App characterizeImage, waiting on task 417 parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 418 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 419 submitted for App calibrate, waiting on task 418 parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 419 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 420 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 420 has memoization hash c5cbe179f08678493596a7fb7cb16665 parsl.dataflow.memoization INFO: Task 420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c480680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/c730bd0e-2f4e-4b34-a01d-35230a9c68e6_isr_898798_9.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/c730bd0e-2f4e-4b34-a01d-35230a9c68e6_isr_898798_9.stderr parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 421 submitted for App characterizeImage, waiting on task 420 parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 421 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 422 submitted for App calibrate, waiting on task 421 parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 422 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 423 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 423 has memoization hash b23cefe75737cfbdf1d8274e65f763ab parsl.dataflow.memoization INFO: Task 423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4809a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/4d4f9af6-8be5-457a-8214-3d0903834ed6_isr_898798_30.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/4d4f9af6-8be5-457a-8214-3d0903834ed6_isr_898798_30.stderr parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 424 submitted for App characterizeImage, waiting on task 423 parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 424 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 425 submitted for App calibrate, waiting on task 424 parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 425 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 426 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 426 has memoization hash 5aeb0134206aa8b6a39a69b9f9909ccb parsl.dataflow.memoization INFO: Task 426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c480cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/a0caebc5-8588-489c-9d43-b55f66962db9_isr_898799_19.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/a0caebc5-8588-489c-9d43-b55f66962db9_isr_898799_19.stderr parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 427 submitted for App characterizeImage, waiting on task 426 parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 427 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 428 submitted for App calibrate, waiting on task 427 parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 428 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 429 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 429 has memoization hash fb12dbf184f0254bda64a035f8153d99 parsl.dataflow.memoization INFO: Task 429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c480fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 139 parsl.dataflow.strategy DEBUG: Executor multi has 139 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 34 more blocks parsl.executors.status_handling INFO: Scaling out by 34 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-13 09:45:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-13 09:45:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 09:45:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/56f892c7-d61d-4ecd-b035-8fce4a816444_isr_898799_33.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/56f892c7-d61d-4ecd-b035-8fce4a816444_isr_898799_33.stderr parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 430 submitted for App characterizeImage, waiting on task 429 parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 430 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 431 submitted for App calibrate, waiting on task 430 parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 431 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 432 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 432 has memoization hash 13dc107ce667610dfb0153b285e05bb0 parsl.dataflow.memoization INFO: Task 432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4814e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/f68b3a1f-661f-4c43-b3dd-00cc3d0e5073_isr_898800_53.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/f68b3a1f-661f-4c43-b3dd-00cc3d0e5073_isr_898800_53.stderr parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 433 submitted for App characterizeImage, waiting on task 432 parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 433 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 434 submitted for App calibrate, waiting on task 433 parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 434 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 435 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 435 has memoization hash 5d01fc3cb81e6db0027c946c893c340f parsl.dataflow.memoization INFO: Task 435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c481800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/dc05af37-e7a7-4587-af35-d1810d260e5e_isr_898695_6.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/dc05af37-e7a7-4587-af35-d1810d260e5e_isr_898695_6.stderr parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 436 submitted for App characterizeImage, waiting on task 435 parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 436 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 437 submitted for App calibrate, waiting on task 436 parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 437 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 438 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 438 has memoization hash c3a0a9682538ac7ef0a675d477618771 parsl.dataflow.memoization INFO: Task 438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c481b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/1bb7ce4e-3cc4-42f1-bc34-f8f68cf3db40_isr_898800_47.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/1bb7ce4e-3cc4-42f1-bc34-f8f68cf3db40_isr_898800_47.stderr parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 439 submitted for App characterizeImage, waiting on task 438 parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 439 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 440 submitted for App calibrate, waiting on task 439 parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 440 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 441 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 441 has memoization hash 235532de60e2ec785d6f78141e23a097 parsl.dataflow.memoization INFO: Task 441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c481e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/7431907f-465a-40ba-b6a1-cbe288898ce3_isr_898798_50.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/7431907f-465a-40ba-b6a1-cbe288898ce3_isr_898798_50.stderr parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 442 submitted for App characterizeImage, waiting on task 441 parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 442 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 443 submitted for App calibrate, waiting on task 442 parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 443 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 444 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 444 has memoization hash eb29a272368cfb93323e5542cb3e6db6 parsl.dataflow.memoization INFO: Task 444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/fc8a0b31-dbd8-4709-8ecb-bfed76ad324c_isr_898798_55.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/fc8a0b31-dbd8-4709-8ecb-bfed76ad324c_isr_898798_55.stderr parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 445 submitted for App characterizeImage, waiting on task 444 parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 445 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 446 submitted for App calibrate, waiting on task 445 parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 446 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 447 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 447 has memoization hash 7693b413523962712cffd017f7a548bd parsl.dataflow.memoization INFO: Task 447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/041488c2-7bd7-4816-9036-d51ab89cb225_isr_898695_43.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/041488c2-7bd7-4816-9036-d51ab89cb225_isr_898695_43.stderr parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 448 submitted for App characterizeImage, waiting on task 447 parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 448 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 449 submitted for App calibrate, waiting on task 448 parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 449 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 450 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 450 has memoization hash c399632e4d15fc60e44f6774315947f6 parsl.dataflow.memoization INFO: Task 450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/ea236db1-82e1-43b0-a35a-827c2848e522_isr_898799_60.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/ea236db1-82e1-43b0-a35a-827c2848e522_isr_898799_60.stderr parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 451 submitted for App characterizeImage, waiting on task 450 parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 451 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 452 submitted for App calibrate, waiting on task 451 parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 452 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 453 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 453 has memoization hash 5dfb31a11ad58dddfab0c20fc9c1241b parsl.dataflow.memoization INFO: Task 453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/c291f742-589b-4187-9f84-070ee3cda67f_isr_898696_48.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/c291f742-589b-4187-9f84-070ee3cda67f_isr_898696_48.stderr parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 454 submitted for App characterizeImage, waiting on task 453 parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 454 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 455 submitted for App calibrate, waiting on task 454 parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 455 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 456 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 456 has memoization hash dd8a8849d14ddee7d0248186598cfe4e parsl.dataflow.memoization INFO: Task 456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/928b569d-efbe-492e-84db-9c6aca2fd956_isr_898798_2.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/928b569d-efbe-492e-84db-9c6aca2fd956_isr_898798_2.stderr parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 457 submitted for App characterizeImage, waiting on task 456 parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 457 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 458 submitted for App calibrate, waiting on task 457 parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 458 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 459 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 459 has memoization hash c1f8a95511dd431112c0341194fcc62d parsl.dataflow.memoization INFO: Task 459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 459 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/86ea0176-2a82-4723-bd29-bb72d84589b2_isr_898695_11.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/86ea0176-2a82-4723-bd29-bb72d84589b2_isr_898695_11.stderr parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 460 submitted for App characterizeImage, waiting on task 459 parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 460 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 461 submitted for App calibrate, waiting on task 460 parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 461 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 462 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 462 has memoization hash 93547fb9205d888c0b29e35336f521d6 parsl.dataflow.memoization INFO: Task 462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/342b8674-1c22-46c5-9afe-0c82f2040cb9_isr_898798_56.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/342b8674-1c22-46c5-9afe-0c82f2040cb9_isr_898798_56.stderr parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 463 submitted for App characterizeImage, waiting on task 462 parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 463 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 464 submitted for App calibrate, waiting on task 463 parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 464 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 465 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 465 has memoization hash 8d4944062fa6c12bfa978ed1b1ac38d0 parsl.dataflow.memoization INFO: Task 465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/535e504c-a5b5-417b-abfa-7f8b72b9af02_isr_898695_10.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/535e504c-a5b5-417b-abfa-7f8b72b9af02_isr_898695_10.stderr parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 466 submitted for App characterizeImage, waiting on task 465 parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 466 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 467 submitted for App calibrate, waiting on task 466 parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 467 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 468 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 468 has memoization hash 028fa278170022c309c106f41fb68115 parsl.dataflow.memoization INFO: Task 468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/816ab556-0529-4a57-9c60-f70b986a281f_isr_898798_31.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/816ab556-0529-4a57-9c60-f70b986a281f_isr_898798_31.stderr parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 469 submitted for App characterizeImage, waiting on task 468 parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 469 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 470 submitted for App calibrate, waiting on task 469 parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 470 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 471 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 471 has memoization hash 362ae681671355166b4a95eeb624471c parsl.dataflow.memoization INFO: Task 471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4825c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/ceb28092-d6c7-40b8-82e8-1bb1426fb50b_isr_898695_14.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/ceb28092-d6c7-40b8-82e8-1bb1426fb50b_isr_898695_14.stderr parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 472 submitted for App characterizeImage, waiting on task 471 parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 472 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 473 submitted for App calibrate, waiting on task 472 parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 473 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 474 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 474 has memoization hash c927f7c3fbaa047d229d30ac94cf305a parsl.dataflow.memoization INFO: Task 474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c482980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/0d9feac5-23e7-4611-966b-0045d66b5282_isr_898800_48.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/0d9feac5-23e7-4611-966b-0045d66b5282_isr_898800_48.stderr parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 475 submitted for App characterizeImage, waiting on task 474 parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 475 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 476 submitted for App calibrate, waiting on task 475 parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 476 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 477 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 477 has memoization hash b850816268ec2ac94bad68ec52558fbf parsl.dataflow.memoization INFO: Task 477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c482ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/6a611205-4a58-4bfa-8d87-37275ece9b31_isr_898798_28.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/6a611205-4a58-4bfa-8d87-37275ece9b31_isr_898798_28.stderr parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 478 submitted for App characterizeImage, waiting on task 477 parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 478 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 479 submitted for App calibrate, waiting on task 478 parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 479 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 480 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 480 has memoization hash c2d3f3d5dba1dfb3b4962f3d10032c94 parsl.dataflow.memoization INFO: Task 480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c482de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/3b960306-2e34-492a-af89-c331228ea9be_isr_898799_59.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/3b960306-2e34-492a-af89-c331228ea9be_isr_898799_59.stderr parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 481 submitted for App characterizeImage, waiting on task 480 parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 481 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 482 submitted for App calibrate, waiting on task 481 parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 482 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 483 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 483 has memoization hash f85bfbf20abf7844941c5369e093d352 parsl.dataflow.memoization INFO: Task 483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c483240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/fe553a02-202f-4e5f-96ed-4cf527f4e7ef_isr_898696_31.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/fe553a02-202f-4e5f-96ed-4cf527f4e7ef_isr_898696_31.stderr parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 484 submitted for App characterizeImage, waiting on task 483 parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 484 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 485 submitted for App calibrate, waiting on task 484 parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 485 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 486 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 486 has memoization hash 4e6b00a5d1b4a8e9793ef3a183ecc2fd parsl.dataflow.memoization INFO: Task 486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c483560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/80ce196e-a08f-447d-943d-1cc22b1fef20_isr_898694_27.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/80ce196e-a08f-447d-943d-1cc22b1fef20_isr_898694_27.stderr parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 487 submitted for App characterizeImage, waiting on task 486 parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 487 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 488 submitted for App calibrate, waiting on task 487 parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 488 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 489 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 489 has memoization hash a7ce022246a5ab6230a9fa7911efc92d parsl.dataflow.memoization INFO: Task 489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4837e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/aaa86e50-3c95-4512-9778-9b7ddd1caa73_isr_898798_62.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/aaa86e50-3c95-4512-9778-9b7ddd1caa73_isr_898798_62.stderr parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 490 submitted for App characterizeImage, waiting on task 489 parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 490 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 491 submitted for App calibrate, waiting on task 490 parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 491 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 492 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 492 has memoization hash 2e39e3f1de2f244c4c38ed4b3fbed4a4 parsl.dataflow.memoization INFO: Task 492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c483ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/55960bf9-aeb3-406f-a9fd-f01f0ec8c7a4_isr_898696_30.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/55960bf9-aeb3-406f-a9fd-f01f0ec8c7a4_isr_898696_30.stderr parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 493 submitted for App characterizeImage, waiting on task 492 parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 493 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 494 submitted for App calibrate, waiting on task 493 parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 494 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 495 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 495 has memoization hash 95ce67f6b0938705d23a89dc4da57b56 parsl.dataflow.memoization INFO: Task 495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c483f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/dd8d0a39-f831-43ee-ae23-317f60f26b98_isr_898695_8.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/dd8d0a39-f831-43ee-ae23-317f60f26b98_isr_898695_8.stderr parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 496 submitted for App characterizeImage, waiting on task 495 parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 496 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 497 submitted for App calibrate, waiting on task 496 parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 497 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 498 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 498 has memoization hash 1c177bfadb1212b0b09e419b82caa537 parsl.dataflow.memoization INFO: Task 498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/8ef8c210-3705-41be-a20e-036e59ebcd34_isr_898799_30.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/8ef8c210-3705-41be-a20e-036e59ebcd34_isr_898799_30.stderr parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 499 submitted for App characterizeImage, waiting on task 498 parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 499 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 500 submitted for App calibrate, waiting on task 499 parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 500 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 501 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 501 has memoization hash 0627e575398cd5ea2a65c7ca1f4d95a9 parsl.dataflow.memoization INFO: Task 501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/aee5f342-a869-4dfb-a1a2-f4d0a95a5b51_isr_898798_58.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/aee5f342-a869-4dfb-a1a2-f4d0a95a5b51_isr_898798_58.stderr parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 502 submitted for App characterizeImage, waiting on task 501 parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 502 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 503 submitted for App calibrate, waiting on task 502 parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 503 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 504 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 504 has memoization hash 3675c1e13b007482af60e6759384eef9 parsl.dataflow.memoization INFO: Task 504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/38fb58fb-7834-4883-8891-792091a6bf30_isr_898800_37.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/38fb58fb-7834-4883-8891-792091a6bf30_isr_898800_37.stderr parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 505 submitted for App characterizeImage, waiting on task 504 parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 505 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 506 submitted for App calibrate, waiting on task 505 parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 506 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 507 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 507 has memoization hash 6b78c98ca00d601fb53163b1b63b4b39 parsl.dataflow.memoization INFO: Task 507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/ca85b22e-7b93-4e92-ab7c-2e2c5794ce98_isr_898799_39.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/ca85b22e-7b93-4e92-ab7c-2e2c5794ce98_isr_898799_39.stderr parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 508 submitted for App characterizeImage, waiting on task 507 parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 508 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 509 submitted for App calibrate, waiting on task 508 parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 509 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 510 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 510 has memoization hash 612c4b65bf79b92b6e00a0bc98e3dc23 parsl.dataflow.memoization INFO: Task 510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/bfcb9197-9596-488e-b2b3-77c75a0f16bd_isr_898799_11.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/bfcb9197-9596-488e-b2b3-77c75a0f16bd_isr_898799_11.stderr parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 511 submitted for App characterizeImage, waiting on task 510 parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 511 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 512 submitted for App calibrate, waiting on task 511 parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 512 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 513 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 513 has memoization hash 8e191d697a1bb542320f9e784c4cfdfb parsl.dataflow.memoization INFO: Task 513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/3e3100ce-0ed8-4622-8b0d-2b8c93307240_isr_898696_53.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/3e3100ce-0ed8-4622-8b0d-2b8c93307240_isr_898696_53.stderr parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 514 submitted for App characterizeImage, waiting on task 513 parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 514 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 515 submitted for App calibrate, waiting on task 514 parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 515 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 516 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 516 has memoization hash 6a8b698c96bf84043a33540699949d12 parsl.dataflow.memoization INFO: Task 516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/88269940-b43f-490c-9ae7-0610e5ae2ae2_isr_898695_56.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/88269940-b43f-490c-9ae7-0610e5ae2ae2_isr_898695_56.stderr parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 517 submitted for App characterizeImage, waiting on task 516 parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 517 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 518 submitted for App calibrate, waiting on task 517 parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 518 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 519 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 519 has memoization hash caf254003b429a427899f2222a8dd6e6 parsl.dataflow.memoization INFO: Task 519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/d41c724e-4b56-41fc-8207-efe268e6fb53_isr_898799_44.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/d41c724e-4b56-41fc-8207-efe268e6fb53_isr_898799_44.stderr parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 520 submitted for App characterizeImage, waiting on task 519 parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 520 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 521 submitted for App calibrate, waiting on task 520 parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 521 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 522 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 522 has memoization hash eef16923819fe152db234d974987f81f parsl.dataflow.memoization INFO: Task 522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/2d26d748-4826-4ac6-b2ac-778f62a73d2e_isr_898696_10.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/2d26d748-4826-4ac6-b2ac-778f62a73d2e_isr_898696_10.stderr parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 523 submitted for App characterizeImage, waiting on task 522 parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 523 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 524 submitted for App calibrate, waiting on task 523 parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 524 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 525 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 525 has memoization hash b923bb61c5e3ac08301708d966e6aea4 parsl.dataflow.memoization INFO: Task 525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/3ef4d5e6-d4fe-4d3e-a9e2-1b9eff80e16b_isr_898694_17.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/3ef4d5e6-d4fe-4d3e-a9e2-1b9eff80e16b_isr_898694_17.stderr parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 526 submitted for App characterizeImage, waiting on task 525 parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 526 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 527 submitted for App calibrate, waiting on task 526 parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 527 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 528 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 528 has memoization hash 8daaef47930933a1692b77599fd9fb8f parsl.dataflow.memoization INFO: Task 528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 528 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/980cfec4-2043-4a92-9ddf-8dd3e22136ce_isr_898696_23.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/980cfec4-2043-4a92-9ddf-8dd3e22136ce_isr_898696_23.stderr parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 529 submitted for App characterizeImage, waiting on task 528 parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 529 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 530 submitted for App calibrate, waiting on task 529 parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 530 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 531 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 531 has memoization hash 17d430884a10802d5dc692449217b3ee parsl.dataflow.memoization INFO: Task 531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/297005f9-8874-4e08-aecd-dcc9bf20c33c_isr_898695_31.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/297005f9-8874-4e08-aecd-dcc9bf20c33c_isr_898695_31.stderr parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 532 submitted for App characterizeImage, waiting on task 531 parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 532 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 533 submitted for App calibrate, waiting on task 532 parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 533 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 534 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 534 has memoization hash 102792a662ecd3f3203092e77baa9178 parsl.dataflow.memoization INFO: Task 534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dc180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/9c233f0e-faab-4b5e-bf6f-ce60b09f7870_isr_898696_29.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/9c233f0e-faab-4b5e-bf6f-ce60b09f7870_isr_898696_29.stderr parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 535 submitted for App characterizeImage, waiting on task 534 parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 535 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 536 submitted for App calibrate, waiting on task 535 parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 536 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 537 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 537 has memoization hash ab7669f960cf6a46be320b53c9ceec92 parsl.dataflow.memoization INFO: Task 537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dc4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/83f138d0-5617-4a8f-b7c9-42f195697f00_isr_898800_32.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/83f138d0-5617-4a8f-b7c9-42f195697f00_isr_898800_32.stderr parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 538 submitted for App characterizeImage, waiting on task 537 parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 538 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 539 submitted for App calibrate, waiting on task 538 parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 539 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 540 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 540 has memoization hash f6bc1f5cd1d944b8d3ba202b95fd107e parsl.dataflow.memoization INFO: Task 540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dc7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/76543e45-d206-4b0e-bb25-a9851eb8a310_isr_898800_20.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/76543e45-d206-4b0e-bb25-a9851eb8a310_isr_898800_20.stderr parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 541 submitted for App characterizeImage, waiting on task 540 parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 541 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 542 submitted for App calibrate, waiting on task 541 parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 542 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 543 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 543 has memoization hash b69360722706345a49ef9c367b887613 parsl.dataflow.memoization INFO: Task 543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dcae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 09:45:09 proc_lsst.multi:170 [INFO] [multi] job_id 17010046 proc_lsst.multi INFO: [multi] job_id 17010046 2024-03-13 09:45:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010046 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010046 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17010046 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-13 09:45:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-13 09:45:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/1b219a5e-8018-4444-9943-992a91c4a17e_isr_898799_48.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/1b219a5e-8018-4444-9943-992a91c4a17e_isr_898799_48.stderr parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 544 submitted for App characterizeImage, waiting on task 543 parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 544 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 545 submitted for App calibrate, waiting on task 544 parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 545 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 546 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 546 has memoization hash cd2fe20119a08d1df1c0f578e886e2e2 parsl.dataflow.memoization INFO: Task 546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dcb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/65882770-57ad-489e-a487-731327dd2e9c_isr_898696_37.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/65882770-57ad-489e-a487-731327dd2e9c_isr_898696_37.stderr parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 547 submitted for App characterizeImage, waiting on task 546 parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 547 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 548 submitted for App calibrate, waiting on task 547 parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 548 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 549 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 549 has memoization hash d9125eaa8789fce4778d61eeb042f288 parsl.dataflow.memoization INFO: Task 549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dcf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/59aa80e5-0751-4754-9395-2094a52c78fe_isr_898799_28.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/59aa80e5-0751-4754-9395-2094a52c78fe_isr_898799_28.stderr parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 550 submitted for App characterizeImage, waiting on task 549 parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 550 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 551 submitted for App calibrate, waiting on task 550 parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 551 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 552 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 552 has memoization hash 304d6d35bca2a228e01686ee32de441f parsl.dataflow.memoization INFO: Task 552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dd260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/09edc1f6-741e-4f21-97e1-d2cce02f0ab4_isr_898800_29.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/09edc1f6-741e-4f21-97e1-d2cce02f0ab4_isr_898800_29.stderr parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 553 submitted for App characterizeImage, waiting on task 552 parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 553 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 554 submitted for App calibrate, waiting on task 553 parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 554 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 555 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 555 has memoization hash d0eda1c5f19a3ba2dffead0f8da82c1c parsl.dataflow.memoization INFO: Task 555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dd580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/c9f49d0d-b8a8-4650-915b-17d37d8f9115_isr_898800_42.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/c9f49d0d-b8a8-4650-915b-17d37d8f9115_isr_898800_42.stderr parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 556 submitted for App characterizeImage, waiting on task 555 parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 556 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 557 submitted for App calibrate, waiting on task 556 parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 557 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 558 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 558 has memoization hash ac26f9915d0b51bbb19939837b4b4200 parsl.dataflow.memoization INFO: Task 558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dd8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 558 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/ce0ca861-207d-4b33-a9f9-ff120490c9fd_isr_898695_59.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/ce0ca861-207d-4b33-a9f9-ff120490c9fd_isr_898695_59.stderr parsl.dataflow.dflow DEBUG: Task 559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 559 submitted for App characterizeImage, waiting on task 558 parsl.dataflow.dflow DEBUG: Task 559 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 559 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 560 submitted for App calibrate, waiting on task 559 parsl.dataflow.dflow DEBUG: Task 560 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 560 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 561 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 561 has memoization hash c48ad17fa68a4ac013d62c374fd2b16d parsl.dataflow.memoization INFO: Task 561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4ddbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 561 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/4843ac7f-c248-4f2b-b658-79730fa7fa60_isr_898694_21.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/4843ac7f-c248-4f2b-b658-79730fa7fa60_isr_898694_21.stderr parsl.dataflow.dflow DEBUG: Task 562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 562 submitted for App characterizeImage, waiting on task 561 parsl.dataflow.dflow DEBUG: Task 562 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 562 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 563 submitted for App calibrate, waiting on task 562 parsl.dataflow.dflow DEBUG: Task 563 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 563 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 564 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 564 has memoization hash 5c9b73179dd38e116a44bc5de1759035 parsl.dataflow.memoization INFO: Task 564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c510040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 564 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/f1ee9a52-c80e-4971-ade2-a8589095180c_isr_898696_17.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/f1ee9a52-c80e-4971-ade2-a8589095180c_isr_898696_17.stderr parsl.dataflow.dflow DEBUG: Task 565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 565 submitted for App characterizeImage, waiting on task 564 parsl.dataflow.dflow DEBUG: Task 565 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 565 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 566 submitted for App calibrate, waiting on task 565 parsl.dataflow.dflow DEBUG: Task 566 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 566 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 567 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 567 has memoization hash 945dc8c4fbf97660854144d9763fba17 parsl.dataflow.memoization INFO: Task 567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c510360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 567 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/9275adc8-1777-45fe-9e66-bb22aed5c3d4_isr_898800_49.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/9275adc8-1777-45fe-9e66-bb22aed5c3d4_isr_898800_49.stderr parsl.dataflow.dflow DEBUG: Task 568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 568 submitted for App characterizeImage, waiting on task 567 parsl.dataflow.dflow DEBUG: Task 568 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 568 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 569 submitted for App calibrate, waiting on task 568 parsl.dataflow.dflow DEBUG: Task 569 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 569 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 570 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 570 has memoization hash efaa72423265df7a63f269f9a5211989 parsl.dataflow.memoization INFO: Task 570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c510680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 570 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/b7d9e3d5-6672-4a15-bb64-4b3f8926db14_isr_898799_26.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/b7d9e3d5-6672-4a15-bb64-4b3f8926db14_isr_898799_26.stderr parsl.dataflow.dflow DEBUG: Task 571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 571 submitted for App characterizeImage, waiting on task 570 parsl.dataflow.dflow DEBUG: Task 571 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 571 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 572 submitted for App calibrate, waiting on task 571 parsl.dataflow.dflow DEBUG: Task 572 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 572 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 573 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 573 has memoization hash 2e1009a0fa174baa05ed1378fd3bff57 parsl.dataflow.memoization INFO: Task 573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4de340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 573 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/9892d11f-0c1c-4c21-8a92-9f89b827b2bf_isr_898798_20.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/9892d11f-0c1c-4c21-8a92-9f89b827b2bf_isr_898798_20.stderr parsl.dataflow.dflow DEBUG: Task 574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 574 submitted for App characterizeImage, waiting on task 573 parsl.dataflow.dflow DEBUG: Task 574 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 574 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 575 submitted for App calibrate, waiting on task 574 parsl.dataflow.dflow DEBUG: Task 575 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 575 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 576 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 576 has memoization hash 5936a8460c3e5221424cf6b47c9ef06b parsl.dataflow.memoization INFO: Task 576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4de660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 576 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/a1a655a6-7cde-4d2e-bfac-40e898026257_isr_898695_16.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/a1a655a6-7cde-4d2e-bfac-40e898026257_isr_898695_16.stderr parsl.dataflow.dflow DEBUG: Task 577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 577 submitted for App characterizeImage, waiting on task 576 parsl.dataflow.dflow DEBUG: Task 577 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 577 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 578 submitted for App calibrate, waiting on task 577 parsl.dataflow.dflow DEBUG: Task 578 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 578 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 579 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 579 has memoization hash 5e982bc6f6135509923514c0d52317f4 parsl.dataflow.memoization INFO: Task 579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4de840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 579 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/070f1e53-fad0-4516-b5d3-38702c819341_isr_898798_60.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/070f1e53-fad0-4516-b5d3-38702c819341_isr_898798_60.stderr parsl.dataflow.dflow DEBUG: Task 580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 580 submitted for App characterizeImage, waiting on task 579 parsl.dataflow.dflow DEBUG: Task 580 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 580 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 581 submitted for App calibrate, waiting on task 580 parsl.dataflow.dflow DEBUG: Task 581 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 581 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 582 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 582 has memoization hash beb0b79893144b6888051408974cfac8 parsl.dataflow.memoization INFO: Task 582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4deb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 582 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/1a3eacb6-b6d7-43dc-aa27-79c3f07a6114_isr_898695_21.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/1a3eacb6-b6d7-43dc-aa27-79c3f07a6114_isr_898695_21.stderr parsl.dataflow.dflow DEBUG: Task 583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 583 submitted for App characterizeImage, waiting on task 582 parsl.dataflow.dflow DEBUG: Task 583 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 583 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 584 submitted for App calibrate, waiting on task 583 parsl.dataflow.dflow DEBUG: Task 584 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 584 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 585 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 585 has memoization hash e02c44ad56d7a32b6a6d33ee443107a0 parsl.dataflow.memoization INFO: Task 585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4defc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 585 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/bfb16a48-d7d4-4733-8ea6-118988daea34_isr_898799_29.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/bfb16a48-d7d4-4733-8ea6-118988daea34_isr_898799_29.stderr parsl.dataflow.dflow DEBUG: Task 586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 586 submitted for App characterizeImage, waiting on task 585 parsl.dataflow.dflow DEBUG: Task 586 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 586 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 587 submitted for App calibrate, waiting on task 586 parsl.dataflow.dflow DEBUG: Task 587 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 587 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 588 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 588 has memoization hash 09f8faa7167bb5ec65bf72cfb333d236 parsl.dataflow.memoization INFO: Task 588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4df2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 588 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/4eb8029a-1caa-4d93-85dc-ca6cd1144f58_isr_898695_24.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/4eb8029a-1caa-4d93-85dc-ca6cd1144f58_isr_898695_24.stderr parsl.dataflow.dflow DEBUG: Task 589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 589 submitted for App characterizeImage, waiting on task 588 parsl.dataflow.dflow DEBUG: Task 589 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 589 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 590 submitted for App calibrate, waiting on task 589 parsl.dataflow.dflow DEBUG: Task 590 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 590 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 591 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 591 has memoization hash b81eafbbd195c45b77d3fb5f2fc825e9 parsl.dataflow.memoization INFO: Task 591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4df560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 591 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/aea39453-e90a-4d09-9dd6-65a8e2b65b98_isr_898798_25.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/aea39453-e90a-4d09-9dd6-65a8e2b65b98_isr_898798_25.stderr parsl.dataflow.dflow DEBUG: Task 592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 592 submitted for App characterizeImage, waiting on task 591 parsl.dataflow.dflow DEBUG: Task 592 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 592 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 593 submitted for App calibrate, waiting on task 592 parsl.dataflow.dflow DEBUG: Task 593 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 593 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 594 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 594 has memoization hash dc9ed16f0c312816559977e48b2c63d8 parsl.dataflow.memoization INFO: Task 594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dfa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 594 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/8411a991-616f-4d1a-a90c-d58598becb1a_isr_898694_45.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/8411a991-616f-4d1a-a90c-d58598becb1a_isr_898694_45.stderr parsl.dataflow.dflow DEBUG: Task 595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 595 submitted for App characterizeImage, waiting on task 594 parsl.dataflow.dflow DEBUG: Task 595 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 595 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 596 submitted for App calibrate, waiting on task 595 parsl.dataflow.dflow DEBUG: Task 596 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 596 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 597 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 597 has memoization hash abaafbdbf45c68f550cd0792ae5ee74f parsl.dataflow.memoization INFO: Task 597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dfce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 597 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/ff482b0f-2509-41fb-bc4d-0cb6b6baec01_isr_898799_41.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/ff482b0f-2509-41fb-bc4d-0cb6b6baec01_isr_898799_41.stderr parsl.dataflow.dflow DEBUG: Task 598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 598 submitted for App characterizeImage, waiting on task 597 parsl.dataflow.dflow DEBUG: Task 598 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 598 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 599 submitted for App calibrate, waiting on task 598 parsl.dataflow.dflow DEBUG: Task 599 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 599 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 600 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 600 has memoization hash 3835a5165ef4674b5a979277a5385892 parsl.dataflow.memoization INFO: Task 600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c5107c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 600 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/0fa6f325-28bf-4d7f-8fbe-71931bef0822_isr_898800_23.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/0fa6f325-28bf-4d7f-8fbe-71931bef0822_isr_898800_23.stderr parsl.dataflow.dflow DEBUG: Task 601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 601 submitted for App characterizeImage, waiting on task 600 parsl.dataflow.dflow DEBUG: Task 601 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 601 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 602 submitted for App calibrate, waiting on task 601 parsl.dataflow.dflow DEBUG: Task 602 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 602 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 603 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 603 has memoization hash ac1e6c35f11ea7b8ecac79bde03cd74d parsl.dataflow.memoization INFO: Task 603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c5109a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 603 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/d1b2154c-f053-4976-92c1-bf22f573a83b_isr_898800_51.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/d1b2154c-f053-4976-92c1-bf22f573a83b_isr_898800_51.stderr parsl.dataflow.dflow DEBUG: Task 604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 604 submitted for App characterizeImage, waiting on task 603 parsl.dataflow.dflow DEBUG: Task 604 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 604 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 605 submitted for App calibrate, waiting on task 604 parsl.dataflow.dflow DEBUG: Task 605 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 605 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 606 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 606 has memoization hash a0e83e0f6ca540b91fa698fe2c7fc716 parsl.dataflow.memoization INFO: Task 606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c510cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 606 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/6b725fec-a4d3-43cd-a74f-ec666a7e3da1_isr_898696_27.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/6b725fec-a4d3-43cd-a74f-ec666a7e3da1_isr_898696_27.stderr parsl.dataflow.dflow DEBUG: Task 607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 607 submitted for App characterizeImage, waiting on task 606 parsl.dataflow.dflow DEBUG: Task 607 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 607 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 608 submitted for App calibrate, waiting on task 607 parsl.dataflow.dflow DEBUG: Task 608 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 608 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 609 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 609 has memoization hash ceb38ea41a21440749c72c65e13970a0 parsl.dataflow.memoization INFO: Task 609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c510fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 609 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/16806378-b168-46d4-8e36-71069f35bd80_isr_898696_60.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/16806378-b168-46d4-8e36-71069f35bd80_isr_898696_60.stderr parsl.dataflow.dflow DEBUG: Task 610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 610 submitted for App characterizeImage, waiting on task 609 parsl.dataflow.dflow DEBUG: Task 610 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 610 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 611 submitted for App calibrate, waiting on task 610 parsl.dataflow.dflow DEBUG: Task 611 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 611 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 612 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 612 has memoization hash 0bc40bcf919347d1af4325c61da814fe parsl.dataflow.memoization INFO: Task 612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c511300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 612 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/95a121b3-6d6b-46d8-a76a-85cf7acef544_isr_898800_30.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/95a121b3-6d6b-46d8-a76a-85cf7acef544_isr_898800_30.stderr parsl.dataflow.dflow DEBUG: Task 613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 613 submitted for App characterizeImage, waiting on task 612 parsl.dataflow.dflow DEBUG: Task 613 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 613 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 614 submitted for App calibrate, waiting on task 613 parsl.dataflow.dflow DEBUG: Task 614 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 614 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 615 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 615 has memoization hash ed3f63875726d4391ea73b3d6aa85478 parsl.dataflow.memoization INFO: Task 615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c511620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 615 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/c27af14e-1b9b-4c54-9036-1644838df165_isr_898695_22.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/c27af14e-1b9b-4c54-9036-1644838df165_isr_898695_22.stderr parsl.dataflow.dflow DEBUG: Task 616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 616 submitted for App characterizeImage, waiting on task 615 parsl.dataflow.dflow DEBUG: Task 616 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 616 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 617 submitted for App calibrate, waiting on task 616 parsl.dataflow.dflow DEBUG: Task 617 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 617 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 618 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 618 has memoization hash df1a068506828b9d56e3e17421aa3182 parsl.dataflow.memoization INFO: Task 618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c511940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 618 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/0b79bf7d-68dc-4c56-83eb-9c764020e18d_isr_898800_38.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/0b79bf7d-68dc-4c56-83eb-9c764020e18d_isr_898800_38.stderr parsl.dataflow.dflow DEBUG: Task 619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 619 submitted for App characterizeImage, waiting on task 618 parsl.dataflow.dflow DEBUG: Task 619 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 619 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 620 submitted for App calibrate, waiting on task 619 parsl.dataflow.dflow DEBUG: Task 620 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 620 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 621 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 621 has memoization hash 3fe5141fe808dee2f7ecaeb087eeca65 parsl.dataflow.memoization INFO: Task 621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c511c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 621 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/fa6d8d5c-54f9-4eac-989c-8d604a1f5ea3_isr_898696_4.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/fa6d8d5c-54f9-4eac-989c-8d604a1f5ea3_isr_898696_4.stderr parsl.dataflow.dflow DEBUG: Task 622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 622 submitted for App characterizeImage, waiting on task 621 parsl.dataflow.dflow DEBUG: Task 622 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 622 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 623 submitted for App calibrate, waiting on task 622 parsl.dataflow.dflow DEBUG: Task 623 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 623 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 624 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 624 has memoization hash 55e59bf14eb4dd50aa9454e65fc8c027 parsl.dataflow.memoization INFO: Task 624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c511f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 624 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/2a294fc0-6d0e-43ca-a89e-d73e8825d5ec_isr_898694_28.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/2a294fc0-6d0e-43ca-a89e-d73e8825d5ec_isr_898694_28.stderr parsl.dataflow.dflow DEBUG: Task 625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 625 submitted for App characterizeImage, waiting on task 624 parsl.dataflow.dflow DEBUG: Task 625 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 625 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 626 submitted for App calibrate, waiting on task 625 parsl.dataflow.dflow DEBUG: Task 626 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 626 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 627 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 627 has memoization hash 0a8b20ee32725ad0c3debda2a0a3cdd0 parsl.dataflow.memoization INFO: Task 627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c5122a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 627 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/137efc9f-66b3-4d9d-b06a-3303abf4e75b_isr_898799_34.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/137efc9f-66b3-4d9d-b06a-3303abf4e75b_isr_898799_34.stderr parsl.dataflow.dflow DEBUG: Task 628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 628 submitted for App characterizeImage, waiting on task 627 parsl.dataflow.dflow DEBUG: Task 628 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 628 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 629 submitted for App calibrate, waiting on task 628 parsl.dataflow.dflow DEBUG: Task 629 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 629 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 630 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 630 has memoization hash e9ad7e935b8267d00ac08144e66fab46 parsl.dataflow.memoization INFO: Task 630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c5125c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 630 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/940eb93e-802d-4282-9afd-95e8a1ce481c_isr_898696_49.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/940eb93e-802d-4282-9afd-95e8a1ce481c_isr_898696_49.stderr parsl.dataflow.dflow DEBUG: Task 631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 631 submitted for App characterizeImage, waiting on task 630 parsl.dataflow.dflow DEBUG: Task 631 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 631 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 632 submitted for App calibrate, waiting on task 631 parsl.dataflow.dflow DEBUG: Task 632 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 632 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 633 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 633 has memoization hash 5241a87baa9e582ff9aef920e6b1b9fa parsl.dataflow.memoization INFO: Task 633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c5128e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 633 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/593418a7-f6e0-46c3-a570-d0090b216e2e_isr_898798_45.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/593418a7-f6e0-46c3-a570-d0090b216e2e_isr_898798_45.stderr parsl.dataflow.dflow DEBUG: Task 634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 634 submitted for App characterizeImage, waiting on task 633 parsl.dataflow.dflow DEBUG: Task 634 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 634 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 635 submitted for App calibrate, waiting on task 634 parsl.dataflow.dflow DEBUG: Task 635 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 635 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 636 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 636 has memoization hash 9042284bde263c62331a6f8969df1e64 parsl.dataflow.memoization INFO: Task 636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c512c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 636 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/37aa4699-3c96-40b8-a467-c42a8d857693_isr_898799_13.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/37aa4699-3c96-40b8-a467-c42a8d857693_isr_898799_13.stderr parsl.dataflow.dflow DEBUG: Task 637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 637 submitted for App characterizeImage, waiting on task 636 parsl.dataflow.dflow DEBUG: Task 637 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 637 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 638 submitted for App calibrate, waiting on task 637 parsl.dataflow.dflow DEBUG: Task 638 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 638 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 639 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 639 has memoization hash a14486862bc70b911b49aab3b2197d79 parsl.dataflow.memoization INFO: Task 639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c512f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 639 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/1d1b85a9-ecd9-4b60-8632-9bdd72cd7961_isr_898799_62.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/1d1b85a9-ecd9-4b60-8632-9bdd72cd7961_isr_898799_62.stderr parsl.dataflow.dflow DEBUG: Task 640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 640 submitted for App characterizeImage, waiting on task 639 parsl.dataflow.dflow DEBUG: Task 640 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 640 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 641 submitted for App calibrate, waiting on task 640 parsl.dataflow.dflow DEBUG: Task 641 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 641 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 642 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 642 has memoization hash ef16fee211ffeb7276ef50cf93712bc7 parsl.dataflow.memoization INFO: Task 642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c513240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 09:45:09 proc_lsst.multi:170 [INFO] [multi] job_id 17010047 proc_lsst.multi INFO: [multi] job_id 17010047 2024-03-13 09:45:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 09:45:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010047 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010047 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17010047 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-13 09:45:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-13 09:45:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 09:45:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 642 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/cb0a0edf-aa75-46ea-874e-4c6902bc94f2_isr_898798_11.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/cb0a0edf-aa75-46ea-874e-4c6902bc94f2_isr_898798_11.stderr parsl.dataflow.dflow DEBUG: Task 643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 643 submitted for App characterizeImage, waiting on task 642 parsl.dataflow.dflow DEBUG: Task 643 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 643 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 644 submitted for App calibrate, waiting on task 643 parsl.dataflow.dflow DEBUG: Task 644 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 644 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 645 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 645 has memoization hash d8a1b3a951df637a24f70dd60332e4ef parsl.dataflow.memoization INFO: Task 645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c5132e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 645 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/65701232-472a-42d1-939d-200a44e08ddd_isr_898799_3.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/65701232-472a-42d1-939d-200a44e08ddd_isr_898799_3.stderr parsl.dataflow.dflow DEBUG: Task 646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 646 submitted for App characterizeImage, waiting on task 645 parsl.dataflow.dflow DEBUG: Task 646 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 646 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 647 submitted for App calibrate, waiting on task 646 parsl.dataflow.dflow DEBUG: Task 647 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 647 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 648 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 648 has memoization hash e75157e0a7030d3eeb3caad2ce304011 parsl.dataflow.memoization INFO: Task 648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c513740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 648 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/31bfde07-ee79-4f9a-9db6-2137cf2c5fa3_isr_898798_18.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/31bfde07-ee79-4f9a-9db6-2137cf2c5fa3_isr_898798_18.stderr parsl.dataflow.dflow DEBUG: Task 649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 649 submitted for App characterizeImage, waiting on task 648 parsl.dataflow.dflow DEBUG: Task 649 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 649 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 650 submitted for App calibrate, waiting on task 649 parsl.dataflow.dflow DEBUG: Task 650 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 650 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 651 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 651 has memoization hash 9c4222227ad83d79970a056362ab4a27 parsl.dataflow.memoization INFO: Task 651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c513a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 651 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/fdbc3fdd-6717-46ea-9a11-5c7e1f93b4ff_isr_898696_40.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/fdbc3fdd-6717-46ea-9a11-5c7e1f93b4ff_isr_898696_40.stderr parsl.dataflow.dflow DEBUG: Task 652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 652 submitted for App characterizeImage, waiting on task 651 parsl.dataflow.dflow DEBUG: Task 652 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 652 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 653 submitted for App calibrate, waiting on task 652 parsl.dataflow.dflow DEBUG: Task 653 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 653 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 654 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 654 has memoization hash ffbf45ac7a69c75c3410595002075e92 parsl.dataflow.memoization INFO: Task 654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c376e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 654 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/d5a1807d-cc06-4855-9b1c-8109e1ce01ba_isr_898800_45.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/d5a1807d-cc06-4855-9b1c-8109e1ce01ba_isr_898800_45.stderr parsl.dataflow.dflow DEBUG: Task 655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 655 submitted for App characterizeImage, waiting on task 654 parsl.dataflow.dflow DEBUG: Task 655 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 655 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 656 submitted for App calibrate, waiting on task 655 parsl.dataflow.dflow DEBUG: Task 656 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 656 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 657 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 657 has memoization hash 03873dcb559624ac487d89ffb01dc6b5 parsl.dataflow.memoization INFO: Task 657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c376b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 657 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/626dba7b-dfa5-4fa1-849c-d3e968e04f7c_isr_898696_22.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/626dba7b-dfa5-4fa1-849c-d3e968e04f7c_isr_898696_22.stderr parsl.dataflow.dflow DEBUG: Task 658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 658 submitted for App characterizeImage, waiting on task 657 parsl.dataflow.dflow DEBUG: Task 658 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 658 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 659 submitted for App calibrate, waiting on task 658 parsl.dataflow.dflow DEBUG: Task 659 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 659 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 660 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 660 has memoization hash 25f52407c51321528cc226c2f6d6c71e parsl.dataflow.memoization INFO: Task 660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c376840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 660 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/718752d7-cb50-41d5-a793-3e89258a2dd9_isr_898694_16.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/718752d7-cb50-41d5-a793-3e89258a2dd9_isr_898694_16.stderr parsl.dataflow.dflow DEBUG: Task 661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 661 submitted for App characterizeImage, waiting on task 660 parsl.dataflow.dflow DEBUG: Task 661 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 661 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 662 submitted for App calibrate, waiting on task 661 parsl.dataflow.dflow DEBUG: Task 662 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 662 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 663 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 663 has memoization hash 49bd97e0e32012407abe24685fc298a4 parsl.dataflow.memoization INFO: Task 663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c376700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 663 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/3195a6fa-d59b-4c56-9815-ab6c46ad69e8_isr_898694_2.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/3195a6fa-d59b-4c56-9815-ab6c46ad69e8_isr_898694_2.stderr parsl.dataflow.dflow DEBUG: Task 664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 664 submitted for App characterizeImage, waiting on task 663 parsl.dataflow.dflow DEBUG: Task 664 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 664 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 665 submitted for App calibrate, waiting on task 664 parsl.dataflow.dflow DEBUG: Task 665 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 665 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 666 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 666 has memoization hash f7e387ebc4ac0049932aed1e131f400c parsl.dataflow.memoization INFO: Task 666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c374220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 666 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/a92c0fa2-6efc-4148-93f4-e8e0e174ca3d_isr_898800_11.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/a92c0fa2-6efc-4148-93f4-e8e0e174ca3d_isr_898800_11.stderr parsl.dataflow.dflow DEBUG: Task 667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 667 submitted for App characterizeImage, waiting on task 666 parsl.dataflow.dflow DEBUG: Task 667 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 667 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 668 submitted for App calibrate, waiting on task 667 parsl.dataflow.dflow DEBUG: Task 668 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 668 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 669 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 669 has memoization hash 4ddeef2b9d2acf2f4796ec26486257c8 parsl.dataflow.memoization INFO: Task 669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c374540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 669 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/435713ce-838b-442b-8f63-90567e447b56_isr_898799_1.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/435713ce-838b-442b-8f63-90567e447b56_isr_898799_1.stderr parsl.dataflow.dflow DEBUG: Task 670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 670 submitted for App characterizeImage, waiting on task 669 parsl.dataflow.dflow DEBUG: Task 670 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 670 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 671 submitted for App calibrate, waiting on task 670 parsl.dataflow.dflow DEBUG: Task 671 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 671 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 672 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 672 has memoization hash c50fb76df17ee4dd471641b15dc9331d parsl.dataflow.memoization INFO: Task 672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c374680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 672 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/d9efea15-1c5e-449b-a9a1-266822831fb2_isr_898694_43.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/d9efea15-1c5e-449b-a9a1-266822831fb2_isr_898694_43.stderr parsl.dataflow.dflow DEBUG: Task 673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 673 submitted for App characterizeImage, waiting on task 672 parsl.dataflow.dflow DEBUG: Task 673 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 673 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 674 submitted for App calibrate, waiting on task 673 parsl.dataflow.dflow DEBUG: Task 674 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 674 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 675 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 675 has memoization hash c28a1c81faf47cd51524834a3c99d0af parsl.dataflow.memoization INFO: Task 675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3749a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 675 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/5b0d054f-72bc-4559-8850-d3278bf4d9e2_isr_898695_1.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/5b0d054f-72bc-4559-8850-d3278bf4d9e2_isr_898695_1.stderr parsl.dataflow.dflow DEBUG: Task 676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 676 submitted for App characterizeImage, waiting on task 675 parsl.dataflow.dflow DEBUG: Task 676 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 676 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 677 submitted for App calibrate, waiting on task 676 parsl.dataflow.dflow DEBUG: Task 677 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 677 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 678 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 678 has memoization hash 6fbc38488043bb33d8659288269f5fb3 parsl.dataflow.memoization INFO: Task 678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c374e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 678 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/19aedd40-ecf7-4fe4-a36c-975cf41e959b_isr_898696_44.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/19aedd40-ecf7-4fe4-a36c-975cf41e959b_isr_898696_44.stderr parsl.dataflow.dflow DEBUG: Task 679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 679 submitted for App characterizeImage, waiting on task 678 parsl.dataflow.dflow DEBUG: Task 679 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 679 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 680 submitted for App calibrate, waiting on task 679 parsl.dataflow.dflow DEBUG: Task 680 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 680 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 681 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 681 has memoization hash 6967e2d3bbcf7b03cf0ed4f5a5bae5c5 parsl.dataflow.memoization INFO: Task 681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c375120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 681 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/b559c7e2-f1eb-41ef-996d-3f7783bd3522_isr_898799_31.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/b559c7e2-f1eb-41ef-996d-3f7783bd3522_isr_898799_31.stderr parsl.dataflow.dflow DEBUG: Task 682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 682 submitted for App characterizeImage, waiting on task 681 parsl.dataflow.dflow DEBUG: Task 682 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 682 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 683 submitted for App calibrate, waiting on task 682 parsl.dataflow.dflow DEBUG: Task 683 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 683 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 684 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 684 has memoization hash 66acb793e2ffd287f5abe6da706ceef2 parsl.dataflow.memoization INFO: Task 684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3753a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 684 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/2183e6d4-2365-4e55-b7e3-00bdb4a7ca1a_isr_898799_50.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/2183e6d4-2365-4e55-b7e3-00bdb4a7ca1a_isr_898799_50.stderr parsl.dataflow.dflow DEBUG: Task 685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 685 submitted for App characterizeImage, waiting on task 684 parsl.dataflow.dflow DEBUG: Task 685 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 685 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 686 submitted for App calibrate, waiting on task 685 parsl.dataflow.dflow DEBUG: Task 686 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 686 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 687 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 687 has memoization hash b2cdfcffdb7c99ea46330ad0dc3dedfe parsl.dataflow.memoization INFO: Task 687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3758a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 687 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/24626fea-563d-4cc3-97ef-4737e7fe36be_isr_898798_29.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/24626fea-563d-4cc3-97ef-4737e7fe36be_isr_898798_29.stderr parsl.dataflow.dflow DEBUG: Task 688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 688 submitted for App characterizeImage, waiting on task 687 parsl.dataflow.dflow DEBUG: Task 688 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 688 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 689 submitted for App calibrate, waiting on task 688 parsl.dataflow.dflow DEBUG: Task 689 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 689 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 690 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 690 has memoization hash 05c4b090d2a2bc5cd6145a47b74ac268 parsl.dataflow.memoization INFO: Task 690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c375b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 690 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/3821b160-ee48-44aa-af6c-d985438f7781_isr_898695_34.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/3821b160-ee48-44aa-af6c-d985438f7781_isr_898695_34.stderr parsl.dataflow.dflow DEBUG: Task 691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 691 submitted for App characterizeImage, waiting on task 690 parsl.dataflow.dflow DEBUG: Task 691 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 691 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 692 submitted for App calibrate, waiting on task 691 parsl.dataflow.dflow DEBUG: Task 692 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 692 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 693 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 693 has memoization hash 65e465f975ffad16a15677f15812906a parsl.dataflow.memoization INFO: Task 693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c376520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 693 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/be4bf9eb-4d4f-4e71-a458-475777ed1931_isr_898694_23.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/be4bf9eb-4d4f-4e71-a458-475777ed1931_isr_898694_23.stderr parsl.dataflow.dflow DEBUG: Task 694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 694 submitted for App characterizeImage, waiting on task 693 parsl.dataflow.dflow DEBUG: Task 694 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 694 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 695 submitted for App calibrate, waiting on task 694 parsl.dataflow.dflow DEBUG: Task 695 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 695 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 696 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 696 has memoization hash 659284bb359d590e56217cbcf121dadc parsl.dataflow.memoization INFO: Task 696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c376200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 696 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/72fbb070-69fc-4fe1-823e-a7b3720505d8_isr_898800_52.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/72fbb070-69fc-4fe1-823e-a7b3720505d8_isr_898800_52.stderr parsl.dataflow.dflow DEBUG: Task 697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 697 submitted for App characterizeImage, waiting on task 696 parsl.dataflow.dflow DEBUG: Task 697 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 697 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 698 submitted for App calibrate, waiting on task 697 parsl.dataflow.dflow DEBUG: Task 698 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 698 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 699 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 699 has memoization hash c51160c3aedf675ce026572b24552362 parsl.dataflow.memoization INFO: Task 699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c375ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 699 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/47692a71-04f2-47db-9589-5bb165d88617_isr_898694_19.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/47692a71-04f2-47db-9589-5bb165d88617_isr_898694_19.stderr parsl.dataflow.dflow DEBUG: Task 700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 700 submitted for App characterizeImage, waiting on task 699 parsl.dataflow.dflow DEBUG: Task 700 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 700 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 701 submitted for App calibrate, waiting on task 700 parsl.dataflow.dflow DEBUG: Task 701 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 701 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 702 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 702 has memoization hash 060a1b00385ac0dc29c33086f4d908a4 parsl.dataflow.memoization INFO: Task 702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c377100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 702 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/ce856a54-5785-46a7-9c51-0dc0c6c9ef98_isr_898798_35.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/ce856a54-5785-46a7-9c51-0dc0c6c9ef98_isr_898798_35.stderr parsl.dataflow.dflow DEBUG: Task 703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 703 submitted for App characterizeImage, waiting on task 702 parsl.dataflow.dflow DEBUG: Task 703 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 703 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 704 submitted for App calibrate, waiting on task 703 parsl.dataflow.dflow DEBUG: Task 704 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 704 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 705 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 705 has memoization hash 0655ecfcf9da675165fa57d80f9f8d42 parsl.dataflow.memoization INFO: Task 705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c377420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 705 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/17215e85-5089-43da-8601-15ef276d3e2f_isr_898696_47.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/17215e85-5089-43da-8601-15ef276d3e2f_isr_898696_47.stderr parsl.dataflow.dflow DEBUG: Task 706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 706 submitted for App characterizeImage, waiting on task 705 parsl.dataflow.dflow DEBUG: Task 706 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 706 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 707 submitted for App calibrate, waiting on task 706 parsl.dataflow.dflow DEBUG: Task 707 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 707 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 708 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 708 has memoization hash f74dd0ed2c9931d1753a5e14228ede22 parsl.dataflow.memoization INFO: Task 708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c377740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 708 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/e3feab52-af24-49e5-b42d-539da8101da9_isr_898696_57.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/e3feab52-af24-49e5-b42d-539da8101da9_isr_898696_57.stderr parsl.dataflow.dflow DEBUG: Task 709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 709 submitted for App characterizeImage, waiting on task 708 parsl.dataflow.dflow DEBUG: Task 709 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 709 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 710 submitted for App calibrate, waiting on task 709 parsl.dataflow.dflow DEBUG: Task 710 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 710 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 711 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 711 has memoization hash 4bc252806c0e5a43a24854e31f2f49af parsl.dataflow.memoization INFO: Task 711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c377a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 711 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/8c72eaab-775b-4cd0-8c1e-73d3b276066b_isr_898798_5.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/8c72eaab-775b-4cd0-8c1e-73d3b276066b_isr_898798_5.stderr parsl.dataflow.dflow DEBUG: Task 712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 712 submitted for App characterizeImage, waiting on task 711 parsl.dataflow.dflow DEBUG: Task 712 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 712 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 713 submitted for App calibrate, waiting on task 712 parsl.dataflow.dflow DEBUG: Task 713 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 713 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 714 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 714 has memoization hash 36d646e08b04505b4ac107ac1bc1784d parsl.dataflow.memoization INFO: Task 714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c377d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 714 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/ad533d91-d02a-4646-9e25-223b381b42b2_isr_898695_5.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/ad533d91-d02a-4646-9e25-223b381b42b2_isr_898695_5.stderr parsl.dataflow.dflow DEBUG: Task 715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 715 submitted for App characterizeImage, waiting on task 714 parsl.dataflow.dflow DEBUG: Task 715 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 715 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 716 submitted for App calibrate, waiting on task 715 parsl.dataflow.dflow DEBUG: Task 716 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 716 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 717 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 717 has memoization hash d5bb36ae0bcae61ef3ae7a7db7144981 parsl.dataflow.memoization INFO: Task 717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 717 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/501cf536-6b25-47f9-a3aa-91a525dda533_isr_898798_39.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/501cf536-6b25-47f9-a3aa-91a525dda533_isr_898798_39.stderr parsl.dataflow.dflow DEBUG: Task 718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 718 submitted for App characterizeImage, waiting on task 717 parsl.dataflow.dflow DEBUG: Task 718 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 718 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 719 submitted for App calibrate, waiting on task 718 parsl.dataflow.dflow DEBUG: Task 719 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 719 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 720 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 720 has memoization hash e482b8ca0ee9ed04e51fbdb79f542c2b parsl.dataflow.memoization INFO: Task 720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 720 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/940aa248-dc89-4e1e-b84a-682d7e63e3fb_isr_898696_6.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/940aa248-dc89-4e1e-b84a-682d7e63e3fb_isr_898696_6.stderr parsl.dataflow.dflow DEBUG: Task 721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 721 submitted for App characterizeImage, waiting on task 720 parsl.dataflow.dflow DEBUG: Task 721 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 721 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 722 submitted for App calibrate, waiting on task 721 parsl.dataflow.dflow DEBUG: Task 722 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 722 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 723 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 723 has memoization hash 498dab46db1527dfe6a2bd95f7e1a093 parsl.dataflow.memoization INFO: Task 723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 723 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/97995ac4-245f-416b-a47b-c33a58e5a192_isr_898694_35.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/97995ac4-245f-416b-a47b-c33a58e5a192_isr_898694_35.stderr parsl.dataflow.dflow DEBUG: Task 724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 724 submitted for App characterizeImage, waiting on task 723 parsl.dataflow.dflow DEBUG: Task 724 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 724 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 725 submitted for App calibrate, waiting on task 724 parsl.dataflow.dflow DEBUG: Task 725 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 725 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 726 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 726 has memoization hash 722b7aa69261420bf4b40a750daf66cd parsl.dataflow.memoization INFO: Task 726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 726 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/2489c1e9-947e-4eea-a270-32ac8ee295bc_isr_898695_57.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/2489c1e9-947e-4eea-a270-32ac8ee295bc_isr_898695_57.stderr parsl.dataflow.dflow DEBUG: Task 727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 727 submitted for App characterizeImage, waiting on task 726 parsl.dataflow.dflow DEBUG: Task 727 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 727 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 728 submitted for App calibrate, waiting on task 727 parsl.dataflow.dflow DEBUG: Task 728 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 728 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 729 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 729 has memoization hash 62a59e494ce84e5f958e64d66a38773a parsl.dataflow.memoization INFO: Task 729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 729 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/cace4c4d-2909-4540-be48-6de128ad9af2_isr_898696_50.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/cace4c4d-2909-4540-be48-6de128ad9af2_isr_898696_50.stderr parsl.dataflow.dflow DEBUG: Task 730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 730 submitted for App characterizeImage, waiting on task 729 parsl.dataflow.dflow DEBUG: Task 730 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 730 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 731 submitted for App calibrate, waiting on task 730 parsl.dataflow.dflow DEBUG: Task 731 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 731 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 732 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 732 has memoization hash cffb3cd0506cd36a41c14bb289eebaf5 parsl.dataflow.memoization INFO: Task 732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 732 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/4e943014-2ceb-40da-950c-7eee1a8bd64a_isr_898696_8.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/4e943014-2ceb-40da-950c-7eee1a8bd64a_isr_898696_8.stderr parsl.dataflow.dflow DEBUG: Task 733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 733 submitted for App characterizeImage, waiting on task 732 parsl.dataflow.dflow DEBUG: Task 733 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 733 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 734 submitted for App calibrate, waiting on task 733 parsl.dataflow.dflow DEBUG: Task 734 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 734 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 735 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 735 has memoization hash fc7bd01c3159b4eb7607675277e1ab91 parsl.dataflow.memoization INFO: Task 735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 735 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/c2b9deea-e73d-44e0-a36b-b8b3d1875615_isr_898696_46.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/c2b9deea-e73d-44e0-a36b-b8b3d1875615_isr_898696_46.stderr parsl.dataflow.dflow DEBUG: Task 736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 736 submitted for App characterizeImage, waiting on task 735 parsl.dataflow.dflow DEBUG: Task 736 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 736 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 737 submitted for App calibrate, waiting on task 736 parsl.dataflow.dflow DEBUG: Task 737 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 737 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 738 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 738 has memoization hash efa70df1e4b04f90673a6bf4d9768504 parsl.dataflow.memoization INFO: Task 738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 738 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/f1911837-c88d-4a24-9fd1-8bfe89fe6bed_isr_898694_49.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/f1911837-c88d-4a24-9fd1-8bfe89fe6bed_isr_898694_49.stderr parsl.dataflow.dflow DEBUG: Task 739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 739 submitted for App characterizeImage, waiting on task 738 parsl.dataflow.dflow DEBUG: Task 739 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 739 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 740 submitted for App calibrate, waiting on task 739 parsl.dataflow.dflow DEBUG: Task 740 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 740 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 741 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 741 has memoization hash 8f8163c8c1e93c20af98b66a21c89537 parsl.dataflow.memoization INFO: Task 741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 09:45:10 proc_lsst.multi:170 [INFO] [multi] job_id 17010048 proc_lsst.multi INFO: [multi] job_id 17010048 2024-03-13 09:45:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 09:45:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010048 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010048 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17010048 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-13 09:45:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-13 09:45:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 09:45:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 741 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/e4491328-430f-4e87-bd88-e6e8f4fd58da_isr_898694_24.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/e4491328-430f-4e87-bd88-e6e8f4fd58da_isr_898694_24.stderr parsl.dataflow.dflow DEBUG: Task 742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 742 submitted for App characterizeImage, waiting on task 741 parsl.dataflow.dflow DEBUG: Task 742 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 742 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 743 submitted for App calibrate, waiting on task 742 parsl.dataflow.dflow DEBUG: Task 743 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 743 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 744 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 744 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 744 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 744 has memoization hash 6aa1ef0fd74ede44ead26e8497dc08f2 parsl.dataflow.memoization INFO: Task 744 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 744 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/f75f1837-1b45-4abf-b78a-a4929e741f1e_isr_898798_47.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/f75f1837-1b45-4abf-b78a-a4929e741f1e_isr_898798_47.stderr parsl.dataflow.dflow DEBUG: Task 745 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 745 submitted for App characterizeImage, waiting on task 744 parsl.dataflow.dflow DEBUG: Task 745 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 745 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 746 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 746 submitted for App calibrate, waiting on task 745 parsl.dataflow.dflow DEBUG: Task 746 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 746 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 747 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 747 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 747 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 747 has memoization hash 1747dedc13bf7382edf1a095f210a0b0 parsl.dataflow.memoization INFO: Task 747 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 747 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/e879ec44-0d3c-47fc-823f-71c504d93541_isr_898696_62.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/e879ec44-0d3c-47fc-823f-71c504d93541_isr_898696_62.stderr parsl.dataflow.dflow DEBUG: Task 748 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 748 submitted for App characterizeImage, waiting on task 747 parsl.dataflow.dflow DEBUG: Task 748 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 748 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 749 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 749 submitted for App calibrate, waiting on task 748 parsl.dataflow.dflow DEBUG: Task 749 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 749 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 750 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 750 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 750 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 750 has memoization hash 3d353042533a5b6e1e3f41611367fe41 parsl.dataflow.memoization INFO: Task 750 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3ba200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 750 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/a47417d9-4462-4cc7-86da-8685e62efefb_isr_898800_43.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/a47417d9-4462-4cc7-86da-8685e62efefb_isr_898800_43.stderr parsl.dataflow.dflow DEBUG: Task 751 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 751 submitted for App characterizeImage, waiting on task 750 parsl.dataflow.dflow DEBUG: Task 751 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 751 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 752 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 752 submitted for App calibrate, waiting on task 751 parsl.dataflow.dflow DEBUG: Task 752 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 752 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 753 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 753 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 753 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 753 has memoization hash 39a6dc14cc93fd2292e9fae44de9cae0 parsl.dataflow.memoization INFO: Task 753 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3ba520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 753 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/fa07517d-faee-454e-aa09-365f78d8f2af_isr_898694_20.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/fa07517d-faee-454e-aa09-365f78d8f2af_isr_898694_20.stderr parsl.dataflow.dflow DEBUG: Task 754 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 754 submitted for App characterizeImage, waiting on task 753 parsl.dataflow.dflow DEBUG: Task 754 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 754 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 755 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 755 submitted for App calibrate, waiting on task 754 parsl.dataflow.dflow DEBUG: Task 755 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 755 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 756 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 756 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 756 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 756 has memoization hash dfcc785b1347b6d50039155b39b202a0 parsl.dataflow.memoization INFO: Task 756 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3ba840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 756 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/f91406e2-cdc0-40e7-91f4-9ca08a6e35f3_isr_898800_39.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/f91406e2-cdc0-40e7-91f4-9ca08a6e35f3_isr_898800_39.stderr parsl.dataflow.dflow DEBUG: Task 757 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 757 submitted for App characterizeImage, waiting on task 756 parsl.dataflow.dflow DEBUG: Task 757 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 757 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 758 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 758 submitted for App calibrate, waiting on task 757 parsl.dataflow.dflow DEBUG: Task 758 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 758 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 759 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 759 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 759 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 759 has memoization hash 5e2d95ff22cd18ca23694bd5e0842c9d parsl.dataflow.memoization INFO: Task 759 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3bab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 759 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/b62f2f07-2e20-4987-9de4-29314abd7104_isr_898799_38.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/b62f2f07-2e20-4987-9de4-29314abd7104_isr_898799_38.stderr parsl.dataflow.dflow DEBUG: Task 760 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 760 submitted for App characterizeImage, waiting on task 759 parsl.dataflow.dflow DEBUG: Task 760 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 760 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 761 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 761 submitted for App calibrate, waiting on task 760 parsl.dataflow.dflow DEBUG: Task 761 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 761 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 762 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 762 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 762 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 762 has memoization hash 1abd421d8fc58c88160d52b6c7131001 parsl.dataflow.memoization INFO: Task 762 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3bade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 762 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/acd945bd-661a-4276-8f96-a4fa1e36e993_isr_898798_49.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/acd945bd-661a-4276-8f96-a4fa1e36e993_isr_898798_49.stderr parsl.dataflow.dflow DEBUG: Task 763 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 763 submitted for App characterizeImage, waiting on task 762 parsl.dataflow.dflow DEBUG: Task 763 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 763 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 764 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 764 submitted for App calibrate, waiting on task 763 parsl.dataflow.dflow DEBUG: Task 764 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 764 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 765 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 765 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 765 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 765 has memoization hash 491ca97a8e2edad31a5694eb1eb6d8d7 parsl.dataflow.memoization INFO: Task 765 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 765 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/22912694-60d5-44e2-9319-ee50f0dad658_isr_898695_32.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/22912694-60d5-44e2-9319-ee50f0dad658_isr_898695_32.stderr parsl.dataflow.dflow DEBUG: Task 766 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 766 submitted for App characterizeImage, waiting on task 765 parsl.dataflow.dflow DEBUG: Task 766 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 766 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 767 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 767 submitted for App calibrate, waiting on task 766 parsl.dataflow.dflow DEBUG: Task 767 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 767 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 768 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 768 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 768 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 768 has memoization hash 53d7093c65826b579b39f96444e0c1c2 parsl.dataflow.memoization INFO: Task 768 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 768 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/41b47ea8-0901-4bd7-9dee-542ea4121eb7_isr_898800_10.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/41b47ea8-0901-4bd7-9dee-542ea4121eb7_isr_898800_10.stderr parsl.dataflow.dflow DEBUG: Task 769 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 769 submitted for App characterizeImage, waiting on task 768 parsl.dataflow.dflow DEBUG: Task 769 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 769 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 770 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 770 submitted for App calibrate, waiting on task 769 parsl.dataflow.dflow DEBUG: Task 770 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 770 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 771 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 771 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 771 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 771 has memoization hash a4f428f15622919609a142218bba0ba5 parsl.dataflow.memoization INFO: Task 771 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 771 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/01b6fb62-742f-4824-ba8f-843fdad73460_isr_898695_7.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/01b6fb62-742f-4824-ba8f-843fdad73460_isr_898695_7.stderr parsl.dataflow.dflow DEBUG: Task 772 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 772 submitted for App characterizeImage, waiting on task 771 parsl.dataflow.dflow DEBUG: Task 772 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 772 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 773 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 773 submitted for App calibrate, waiting on task 772 parsl.dataflow.dflow DEBUG: Task 773 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 773 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 774 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 774 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 774 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 774 has memoization hash 79c0c2dfd064745efa9e76d427f5fdcc parsl.dataflow.memoization INFO: Task 774 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 774 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/3c6995ab-224c-4226-bb91-c755fd377c45_isr_898694_4.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/3c6995ab-224c-4226-bb91-c755fd377c45_isr_898694_4.stderr parsl.dataflow.dflow DEBUG: Task 775 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 775 submitted for App characterizeImage, waiting on task 774 parsl.dataflow.dflow DEBUG: Task 775 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 775 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 776 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 776 submitted for App calibrate, waiting on task 775 parsl.dataflow.dflow DEBUG: Task 776 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 776 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 777 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 777 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 777 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 777 has memoization hash 69ba715227023c8b6911a853033280ba parsl.dataflow.memoization INFO: Task 777 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 777 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/1d88ffb0-bbbc-41f4-b7f9-e1788dd925de_isr_898800_13.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/1d88ffb0-bbbc-41f4-b7f9-e1788dd925de_isr_898800_13.stderr parsl.dataflow.dflow DEBUG: Task 778 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 778 submitted for App characterizeImage, waiting on task 777 parsl.dataflow.dflow DEBUG: Task 778 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 778 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 779 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 779 submitted for App calibrate, waiting on task 778 parsl.dataflow.dflow DEBUG: Task 779 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 779 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 780 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 780 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 780 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 780 has memoization hash 5ecc893e023f8f6bdb3e99e150c83dad parsl.dataflow.memoization INFO: Task 780 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 780 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/13fab25c-8b2d-4b86-9b2c-3f8b7d768958_isr_898800_28.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/13fab25c-8b2d-4b86-9b2c-3f8b7d768958_isr_898800_28.stderr parsl.dataflow.dflow DEBUG: Task 781 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 781 submitted for App characterizeImage, waiting on task 780 parsl.dataflow.dflow DEBUG: Task 781 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 781 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 782 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 782 submitted for App calibrate, waiting on task 781 parsl.dataflow.dflow DEBUG: Task 782 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 782 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 783 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 783 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 783 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 783 has memoization hash b74ad18ccfa241c1a297507976655c13 parsl.dataflow.memoization INFO: Task 783 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 783 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/75630e31-b7fd-4960-9d24-7972f3439a70_isr_898696_32.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/75630e31-b7fd-4960-9d24-7972f3439a70_isr_898696_32.stderr parsl.dataflow.dflow DEBUG: Task 784 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 784 submitted for App characterizeImage, waiting on task 783 parsl.dataflow.dflow DEBUG: Task 784 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 784 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 785 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 785 submitted for App calibrate, waiting on task 784 parsl.dataflow.dflow DEBUG: Task 785 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 785 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 786 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 786 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 786 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 786 has memoization hash 21338850196b658cea5dc3c4bb755389 parsl.dataflow.memoization INFO: Task 786 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3bb560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 786 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/4b4ecef7-fe87-458e-a25b-036ce1427113_isr_898799_16.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/4b4ecef7-fe87-458e-a25b-036ce1427113_isr_898799_16.stderr parsl.dataflow.dflow DEBUG: Task 787 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 787 submitted for App characterizeImage, waiting on task 786 parsl.dataflow.dflow DEBUG: Task 787 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 787 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 788 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 788 submitted for App calibrate, waiting on task 787 parsl.dataflow.dflow DEBUG: Task 788 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 788 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 789 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 789 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 789 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 789 has memoization hash 3a12c4c5cb399ac9249e68f515a43257 parsl.dataflow.memoization INFO: Task 789 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3bb920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 789 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/b42611e2-287a-4baa-b856-f2975e06b01d_isr_898799_61.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/b42611e2-287a-4baa-b856-f2975e06b01d_isr_898799_61.stderr parsl.dataflow.dflow DEBUG: Task 790 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 790 submitted for App characterizeImage, waiting on task 789 parsl.dataflow.dflow DEBUG: Task 790 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 790 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 791 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 791 submitted for App calibrate, waiting on task 790 parsl.dataflow.dflow DEBUG: Task 791 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 791 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 792 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 792 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 792 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 792 has memoization hash b2b856db7679d874cc1b88d9ef7ee993 parsl.dataflow.memoization INFO: Task 792 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3bba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 792 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/e23ec1b5-0a07-455d-9fd5-634e715fea86_isr_898800_44.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/e23ec1b5-0a07-455d-9fd5-634e715fea86_isr_898800_44.stderr parsl.dataflow.dflow DEBUG: Task 793 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 793 submitted for App characterizeImage, waiting on task 792 parsl.dataflow.dflow DEBUG: Task 793 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 793 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 794 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 794 submitted for App calibrate, waiting on task 793 parsl.dataflow.dflow DEBUG: Task 794 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 794 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 795 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 795 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 795 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 795 has memoization hash 157e2707f5f549fd2011f376297d4f14 parsl.dataflow.memoization INFO: Task 795 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3bbd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 795 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/d7f97609-9225-4aa5-9bf8-359c0eb4385d_isr_898798_36.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/d7f97609-9225-4aa5-9bf8-359c0eb4385d_isr_898798_36.stderr parsl.dataflow.dflow DEBUG: Task 796 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 796 submitted for App characterizeImage, waiting on task 795 parsl.dataflow.dflow DEBUG: Task 796 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 796 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 797 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 797 submitted for App calibrate, waiting on task 796 parsl.dataflow.dflow DEBUG: Task 797 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 797 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 798 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 798 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 798 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 798 has memoization hash 38c887e918a8b32a37cc385e0fbe308e parsl.dataflow.memoization INFO: Task 798 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 798 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/1e771db2-bde5-4cf8-8db9-59c6e9df85f8_isr_898800_19.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/1e771db2-bde5-4cf8-8db9-59c6e9df85f8_isr_898800_19.stderr parsl.dataflow.dflow DEBUG: Task 799 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 799 submitted for App characterizeImage, waiting on task 798 parsl.dataflow.dflow DEBUG: Task 799 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 799 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 800 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 800 submitted for App calibrate, waiting on task 799 parsl.dataflow.dflow DEBUG: Task 800 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 800 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 801 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 801 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 801 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 801 has memoization hash d7b4c641405d2ce0962d863b3b0a99ab parsl.dataflow.memoization INFO: Task 801 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 801 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/7c10280c-45f9-4b24-a9b9-fffc88e35481_isr_898694_32.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/7c10280c-45f9-4b24-a9b9-fffc88e35481_isr_898694_32.stderr parsl.dataflow.dflow DEBUG: Task 802 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 802 submitted for App characterizeImage, waiting on task 801 parsl.dataflow.dflow DEBUG: Task 802 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 802 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 803 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 803 submitted for App calibrate, waiting on task 802 parsl.dataflow.dflow DEBUG: Task 803 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 803 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 804 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 804 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 804 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 804 has memoization hash 825cf1c4f7838ec4ec35bd78ece16f4b parsl.dataflow.memoization INFO: Task 804 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 804 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/6088d40c-3ed0-4a24-b7e4-088088cbfc8b_isr_898799_54.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/6088d40c-3ed0-4a24-b7e4-088088cbfc8b_isr_898799_54.stderr parsl.dataflow.dflow DEBUG: Task 805 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 805 submitted for App characterizeImage, waiting on task 804 parsl.dataflow.dflow DEBUG: Task 805 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 805 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 806 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 806 submitted for App calibrate, waiting on task 805 parsl.dataflow.dflow DEBUG: Task 806 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 806 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 807 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 807 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 807 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 807 has memoization hash c85a451dc474193bc4cb696ef9948f95 parsl.dataflow.memoization INFO: Task 807 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 807 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/fa1b9937-03fe-4fe2-a634-0199ccb50687_isr_898799_55.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/fa1b9937-03fe-4fe2-a634-0199ccb50687_isr_898799_55.stderr parsl.dataflow.dflow DEBUG: Task 808 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 808 submitted for App characterizeImage, waiting on task 807 parsl.dataflow.dflow DEBUG: Task 808 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 808 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 809 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 809 submitted for App calibrate, waiting on task 808 parsl.dataflow.dflow DEBUG: Task 809 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 809 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 810 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 810 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 810 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 810 has memoization hash 6df8c030a0b1b62db25263cdf81ec8c8 parsl.dataflow.memoization INFO: Task 810 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 810 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/0833e567-18e2-4f4e-b81b-58eb68ee0b67_isr_898799_17.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/0833e567-18e2-4f4e-b81b-58eb68ee0b67_isr_898799_17.stderr parsl.dataflow.dflow DEBUG: Task 811 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 811 submitted for App characterizeImage, waiting on task 810 parsl.dataflow.dflow DEBUG: Task 811 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 811 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 812 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 812 submitted for App calibrate, waiting on task 811 parsl.dataflow.dflow DEBUG: Task 812 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 812 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 813 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 813 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 813 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 813 has memoization hash cd11bb7d230e539f6e42b968d092aa79 parsl.dataflow.memoization INFO: Task 813 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 813 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/2cc4d335-6b6c-4958-ab9e-559289881311_isr_898799_21.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/2cc4d335-6b6c-4958-ab9e-559289881311_isr_898799_21.stderr parsl.dataflow.dflow DEBUG: Task 814 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 814 submitted for App characterizeImage, waiting on task 813 parsl.dataflow.dflow DEBUG: Task 814 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 814 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 815 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 815 submitted for App calibrate, waiting on task 814 parsl.dataflow.dflow DEBUG: Task 815 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 815 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 816 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 816 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 816 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 816 has memoization hash 6241e8dfaef1cdca537fddeee631ae10 parsl.dataflow.memoization INFO: Task 816 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 816 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/280a5be1-369a-485a-8cc7-c005e02be30f_isr_898799_35.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/280a5be1-369a-485a-8cc7-c005e02be30f_isr_898799_35.stderr parsl.dataflow.dflow DEBUG: Task 817 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 817 submitted for App characterizeImage, waiting on task 816 parsl.dataflow.dflow DEBUG: Task 817 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 817 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 818 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 818 submitted for App calibrate, waiting on task 817 parsl.dataflow.dflow DEBUG: Task 818 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 818 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 819 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 819 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 819 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 819 has memoization hash 2397d361a6060eb24bfe566829b183e8 parsl.dataflow.memoization INFO: Task 819 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 819 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/7c794282-6c86-4217-874b-99acab999514_isr_898696_51.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/7c794282-6c86-4217-874b-99acab999514_isr_898696_51.stderr parsl.dataflow.dflow DEBUG: Task 820 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 820 submitted for App characterizeImage, waiting on task 819 parsl.dataflow.dflow DEBUG: Task 820 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 820 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 821 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 821 submitted for App calibrate, waiting on task 820 parsl.dataflow.dflow DEBUG: Task 821 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 821 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 822 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 822 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 822 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 822 has memoization hash 96b8812222f8c85914cb428929cd09a2 parsl.dataflow.memoization INFO: Task 822 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 822 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/2280e2d9-565c-4f49-8e8d-f7c781bbaba1_isr_898800_17.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/2280e2d9-565c-4f49-8e8d-f7c781bbaba1_isr_898800_17.stderr parsl.dataflow.dflow DEBUG: Task 823 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 823 submitted for App characterizeImage, waiting on task 822 parsl.dataflow.dflow DEBUG: Task 823 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 823 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 824 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 824 submitted for App calibrate, waiting on task 823 parsl.dataflow.dflow DEBUG: Task 824 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 824 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 825 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 825 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 825 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 825 has memoization hash d18791b0eaf55bb7451bfeca5072d4f5 parsl.dataflow.memoization INFO: Task 825 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 825 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/645d3164-4768-43a3-9c07-10a42976d8c1_isr_898696_39.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/645d3164-4768-43a3-9c07-10a42976d8c1_isr_898696_39.stderr parsl.dataflow.dflow DEBUG: Task 826 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 826 submitted for App characterizeImage, waiting on task 825 parsl.dataflow.dflow DEBUG: Task 826 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 826 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 827 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 827 submitted for App calibrate, waiting on task 826 parsl.dataflow.dflow DEBUG: Task 827 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 827 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 828 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 828 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 828 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 828 has memoization hash 441c2100b9d566354feaadf06d0329ac parsl.dataflow.memoization INFO: Task 828 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 828 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/c85ec6d2-df10-4915-8ed0-f3ff55499d1e_isr_898694_34.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/c85ec6d2-df10-4915-8ed0-f3ff55499d1e_isr_898694_34.stderr parsl.dataflow.dflow DEBUG: Task 829 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 829 submitted for App characterizeImage, waiting on task 828 parsl.dataflow.dflow DEBUG: Task 829 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 829 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 830 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 830 submitted for App calibrate, waiting on task 829 parsl.dataflow.dflow DEBUG: Task 830 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 830 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 831 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 831 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 831 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 831 has memoization hash de240c94aea65e52d0e43369ae0299d0 parsl.dataflow.memoization INFO: Task 831 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 831 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/3c8c2dc4-b7cd-479b-bdda-3e4645d2e91c_isr_898694_26.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/3c8c2dc4-b7cd-479b-bdda-3e4645d2e91c_isr_898694_26.stderr parsl.dataflow.dflow DEBUG: Task 832 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 832 submitted for App characterizeImage, waiting on task 831 parsl.dataflow.dflow DEBUG: Task 832 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 832 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 833 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 833 submitted for App calibrate, waiting on task 832 parsl.dataflow.dflow DEBUG: Task 833 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 833 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 834 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 834 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 834 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 834 has memoization hash 72d956b0374cfdb4af2a146281a192b1 parsl.dataflow.memoization INFO: Task 834 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 834 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/d425a1e1-3548-43f7-8ad9-9c5d104f3618_isr_898799_40.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/d425a1e1-3548-43f7-8ad9-9c5d104f3618_isr_898799_40.stderr parsl.dataflow.dflow DEBUG: Task 835 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 835 submitted for App characterizeImage, waiting on task 834 parsl.dataflow.dflow DEBUG: Task 835 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 835 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 836 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 836 submitted for App calibrate, waiting on task 835 parsl.dataflow.dflow DEBUG: Task 836 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 836 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 837 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 837 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 837 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 837 has memoization hash 821d7416a807a19597cc1cccfa4d43ba parsl.dataflow.memoization INFO: Task 837 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 837 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/adc5886a-c52d-4669-88c7-46407040e294_isr_898798_44.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/adc5886a-c52d-4669-88c7-46407040e294_isr_898798_44.stderr parsl.dataflow.dflow DEBUG: Task 838 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 838 submitted for App characterizeImage, waiting on task 837 parsl.dataflow.dflow DEBUG: Task 838 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 838 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 839 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 839 submitted for App calibrate, waiting on task 838 parsl.dataflow.dflow DEBUG: Task 839 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 839 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 840 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 840 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 840 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 840 has memoization hash b7fd676fa0659213a050229c6f92b700 parsl.dataflow.memoization INFO: Task 840 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c440180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 840 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/7b2152f8-3942-4a6b-a237-324d28b51a91_isr_898696_14.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/7b2152f8-3942-4a6b-a237-324d28b51a91_isr_898696_14.stderr parsl.dataflow.dflow DEBUG: Task 841 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 841 submitted for App characterizeImage, waiting on task 840 parsl.dataflow.dflow DEBUG: Task 841 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 841 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 842 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 842 submitted for App calibrate, waiting on task 841 parsl.dataflow.dflow DEBUG: Task 842 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 842 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 843 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 843 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 843 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 843 has memoization hash 337b311920e2566c9ad6cfbf1ebf3dff parsl.dataflow.memoization INFO: Task 843 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4404a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 843 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/14a96df3-3d35-47e3-b334-2461603716ed_isr_898798_42.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/14a96df3-3d35-47e3-b334-2461603716ed_isr_898798_42.stderr parsl.dataflow.dflow DEBUG: Task 844 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 844 submitted for App characterizeImage, waiting on task 843 parsl.dataflow.dflow DEBUG: Task 844 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 844 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 845 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 845 submitted for App calibrate, waiting on task 844 parsl.dataflow.dflow DEBUG: Task 845 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 845 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 846 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 846 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 846 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 846 has memoization hash 246d1d17d9107257219ed329fe0f3df6 parsl.dataflow.memoization INFO: Task 846 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4407c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 846 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/ae6440d1-2994-4b33-8258-56d50cac5c56_isr_898695_52.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/ae6440d1-2994-4b33-8258-56d50cac5c56_isr_898695_52.stderr parsl.dataflow.dflow DEBUG: Task 847 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 847 submitted for App characterizeImage, waiting on task 846 parsl.dataflow.dflow DEBUG: Task 847 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 847 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 848 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 848 submitted for App calibrate, waiting on task 847 parsl.dataflow.dflow DEBUG: Task 848 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 848 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 849 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 849 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 849 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 849 has memoization hash 094e566f6134a93f4977590a12fbd236 parsl.dataflow.memoization INFO: Task 849 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c440ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 849 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/5d061782-5758-47a5-9150-c00756cd10dc_isr_898799_56.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/5d061782-5758-47a5-9150-c00756cd10dc_isr_898799_56.stderr parsl.dataflow.dflow DEBUG: Task 850 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 850 submitted for App characterizeImage, waiting on task 849 parsl.dataflow.dflow DEBUG: Task 850 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 850 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 851 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 851 submitted for App calibrate, waiting on task 850 parsl.dataflow.dflow DEBUG: Task 851 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 851 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 852 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 852 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 852 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 852 has memoization hash 2d4f96b42ee8bd372e508af48a5550f0 parsl.dataflow.memoization INFO: Task 852 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c440e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 852 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/6db1ffd4-daf9-4063-bd0a-5cc470744ae9_isr_898694_8.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/6db1ffd4-daf9-4063-bd0a-5cc470744ae9_isr_898694_8.stderr parsl.dataflow.dflow DEBUG: Task 853 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 853 submitted for App characterizeImage, waiting on task 852 parsl.dataflow.dflow DEBUG: Task 853 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 853 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 854 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 854 submitted for App calibrate, waiting on task 853 parsl.dataflow.dflow DEBUG: Task 854 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 854 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 855 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 855 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 855 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 855 has memoization hash 708bb8e0c4909af25b3da6d17c287c01 parsl.dataflow.memoization INFO: Task 855 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c441120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 855 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/65262dfb-3d51-4170-b32b-5dfc40f4ea87_isr_898694_51.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/65262dfb-3d51-4170-b32b-5dfc40f4ea87_isr_898694_51.stderr parsl.dataflow.dflow DEBUG: Task 856 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 856 submitted for App characterizeImage, waiting on task 855 parsl.dataflow.dflow DEBUG: Task 856 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 856 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 857 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 857 submitted for App calibrate, waiting on task 856 parsl.dataflow.dflow DEBUG: Task 857 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 857 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 858 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 858 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 858 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 858 has memoization hash 9d5b9f4bfeddb1aec50214ed0c7ee061 parsl.dataflow.memoization INFO: Task 858 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c441440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 858 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/45c3ca67-b6d6-4b45-9214-584f4b64dbee_isr_898695_30.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/45c3ca67-b6d6-4b45-9214-584f4b64dbee_isr_898695_30.stderr parsl.dataflow.dflow DEBUG: Task 859 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 859 submitted for App characterizeImage, waiting on task 858 parsl.dataflow.dflow DEBUG: Task 859 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 859 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 860 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 860 submitted for App calibrate, waiting on task 859 parsl.dataflow.dflow DEBUG: Task 860 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 860 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 861 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 861 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 861 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 861 has memoization hash c6ef58002c2537e20851802f4ab2fc5a parsl.dataflow.memoization INFO: Task 861 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c441760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 861 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/8a51d2ab-c486-4aa8-b964-432e3ba688d0_isr_898696_7.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/8a51d2ab-c486-4aa8-b964-432e3ba688d0_isr_898696_7.stderr parsl.dataflow.dflow DEBUG: Task 862 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 862 submitted for App characterizeImage, waiting on task 861 parsl.dataflow.dflow DEBUG: Task 862 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 862 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 863 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 863 submitted for App calibrate, waiting on task 862 parsl.dataflow.dflow DEBUG: Task 863 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 863 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 864 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 864 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 864 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 864 has memoization hash 339a5000edd627e46c2d6e6631f62cdd parsl.dataflow.memoization INFO: Task 864 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c441a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 864 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/cb217243-9466-46a6-ad0b-680f291880d2_isr_898694_30.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/cb217243-9466-46a6-ad0b-680f291880d2_isr_898694_30.stderr parsl.dataflow.dflow DEBUG: Task 865 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 865 submitted for App characterizeImage, waiting on task 864 parsl.dataflow.dflow DEBUG: Task 865 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 865 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 866 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 866 submitted for App calibrate, waiting on task 865 parsl.dataflow.dflow DEBUG: Task 866 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 866 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 867 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 867 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 867 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 867 has memoization hash 7a866cde8379499ce1d060e6226b6e9b parsl.dataflow.memoization INFO: Task 867 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c441da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 867 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/7e6a9095-e51f-41e1-8e85-2ee6778c8710_isr_898694_11.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/7e6a9095-e51f-41e1-8e85-2ee6778c8710_isr_898694_11.stderr parsl.dataflow.dflow DEBUG: Task 868 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 868 submitted for App characterizeImage, waiting on task 867 parsl.dataflow.dflow DEBUG: Task 868 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 868 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 869 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 869 submitted for App calibrate, waiting on task 868 parsl.dataflow.dflow DEBUG: Task 869 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 869 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 870 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 870 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 870 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 870 has memoization hash 8e9d4747ccad2e86b387c758bb67b0bf parsl.dataflow.memoization INFO: Task 870 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4420c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 870 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/f1c4813a-5180-4788-a641-a5180d1fa9bd_isr_898696_54.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/f1c4813a-5180-4788-a641-a5180d1fa9bd_isr_898696_54.stderr parsl.dataflow.dflow DEBUG: Task 871 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 871 submitted for App characterizeImage, waiting on task 870 parsl.dataflow.dflow DEBUG: Task 871 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 871 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 872 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 872 submitted for App calibrate, waiting on task 871 parsl.dataflow.dflow DEBUG: Task 872 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 872 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 873 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 873 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 873 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 873 has memoization hash 45df10f7cdb032381ba6a3e8770d9a57 parsl.dataflow.memoization INFO: Task 873 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4423e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 873 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/6bcc0346-1294-4936-ae91-9243f273f472_isr_898694_37.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/6bcc0346-1294-4936-ae91-9243f273f472_isr_898694_37.stderr parsl.dataflow.dflow DEBUG: Task 874 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 874 submitted for App characterizeImage, waiting on task 873 parsl.dataflow.dflow DEBUG: Task 874 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 874 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 875 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 875 submitted for App calibrate, waiting on task 874 parsl.dataflow.dflow DEBUG: Task 875 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 875 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 876 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 876 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 876 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 876 has memoization hash 8d6a6a0ea0320652a32a9a2fe7e1f68d parsl.dataflow.memoization INFO: Task 876 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c442700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 876 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/4b3933b2-7b06-417b-85ae-718c8ed1c25b_isr_898696_25.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/4b3933b2-7b06-417b-85ae-718c8ed1c25b_isr_898696_25.stderr parsl.dataflow.dflow DEBUG: Task 877 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 877 submitted for App characterizeImage, waiting on task 876 parsl.dataflow.dflow DEBUG: Task 877 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 877 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 878 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 878 submitted for App calibrate, waiting on task 877 parsl.dataflow.dflow DEBUG: Task 878 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 878 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 879 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 879 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 879 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 879 has memoization hash d444490d4ec3699f92444658c851a6ce parsl.dataflow.memoization INFO: Task 879 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c442a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 879 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/b66bf1f5-5ec7-4775-9851-a3e10a4ba270_isr_898798_3.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/b66bf1f5-5ec7-4775-9851-a3e10a4ba270_isr_898798_3.stderr parsl.dataflow.dflow DEBUG: Task 880 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 880 submitted for App characterizeImage, waiting on task 879 parsl.dataflow.dflow DEBUG: Task 880 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 880 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 881 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 881 submitted for App calibrate, waiting on task 880 parsl.dataflow.dflow DEBUG: Task 881 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 881 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 882 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 882 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 882 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 882 has memoization hash 5e990a55c07a0599231c92d22305a91c parsl.dataflow.memoization INFO: Task 882 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c442d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 882 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/6f265980-2235-4801-ab94-c436bf47de23_isr_898798_22.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/6f265980-2235-4801-ab94-c436bf47de23_isr_898798_22.stderr parsl.dataflow.dflow DEBUG: Task 883 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 883 submitted for App characterizeImage, waiting on task 882 parsl.dataflow.dflow DEBUG: Task 883 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 883 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 884 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 884 submitted for App calibrate, waiting on task 883 parsl.dataflow.dflow DEBUG: Task 884 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 884 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 885 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 885 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 885 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 885 has memoization hash c644e2ac49b20256f9468bd38272a69a parsl.dataflow.memoization INFO: Task 885 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c443060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 885 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/ae1db5e7-3eeb-4b3e-b50d-492fc6c37434_isr_898799_14.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/ae1db5e7-3eeb-4b3e-b50d-492fc6c37434_isr_898799_14.stderr parsl.dataflow.dflow DEBUG: Task 886 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 886 submitted for App characterizeImage, waiting on task 885 parsl.dataflow.dflow DEBUG: Task 886 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 886 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 887 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 887 submitted for App calibrate, waiting on task 886 parsl.dataflow.dflow DEBUG: Task 887 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 887 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 888 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 888 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 888 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 888 has memoization hash b91e73cac9294c69d79d319f8094a3b9 parsl.dataflow.memoization INFO: Task 888 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c443380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 888 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/06b0a523-ab6b-43c0-9741-339e4df37853_isr_898694_52.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/06b0a523-ab6b-43c0-9741-339e4df37853_isr_898694_52.stderr parsl.dataflow.dflow DEBUG: Task 889 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 889 submitted for App characterizeImage, waiting on task 888 parsl.dataflow.dflow DEBUG: Task 889 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 889 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 890 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 890 submitted for App calibrate, waiting on task 889 parsl.dataflow.dflow DEBUG: Task 890 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 890 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 891 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 891 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 891 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 891 has memoization hash 134a81519f7fd7e6e6c0f582be474ac3 parsl.dataflow.memoization INFO: Task 891 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4436a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 891 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/4ebebbde-91fa-4525-aa09-3aeda3acb51f_isr_898798_59.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/4ebebbde-91fa-4525-aa09-3aeda3acb51f_isr_898798_59.stderr parsl.dataflow.dflow DEBUG: Task 892 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 892 submitted for App characterizeImage, waiting on task 891 parsl.dataflow.dflow DEBUG: Task 892 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 892 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 893 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 893 submitted for App calibrate, waiting on task 892 parsl.dataflow.dflow DEBUG: Task 893 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 893 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 894 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 894 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 894 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 894 has memoization hash baaa506d45dec6e18793892b4a64196a parsl.dataflow.memoization INFO: Task 894 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4439c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 894 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/078acad5-b275-4475-a1e6-a88236b122b8_isr_898798_34.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/078acad5-b275-4475-a1e6-a88236b122b8_isr_898798_34.stderr parsl.dataflow.dflow DEBUG: Task 895 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 895 submitted for App characterizeImage, waiting on task 894 parsl.dataflow.dflow DEBUG: Task 895 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 895 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 896 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 896 submitted for App calibrate, waiting on task 895 parsl.dataflow.dflow DEBUG: Task 896 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 896 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 897 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 897 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 897 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 897 has memoization hash 29df56e58bc6c40ec47039f749da6064 parsl.dataflow.memoization INFO: Task 897 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c443ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 897 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/ee1d8302-0ca1-43d2-9d0c-7725a7328f9c_isr_898800_56.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/ee1d8302-0ca1-43d2-9d0c-7725a7328f9c_isr_898800_56.stderr parsl.dataflow.dflow DEBUG: Task 898 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 898 submitted for App characterizeImage, waiting on task 897 parsl.dataflow.dflow DEBUG: Task 898 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 898 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 899 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 899 submitted for App calibrate, waiting on task 898 parsl.dataflow.dflow DEBUG: Task 899 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 899 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 900 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 900 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 900 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 900 has memoization hash 9141156de10cd5160a8069ba29800e0f parsl.dataflow.memoization INFO: Task 900 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c290040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 09:45:10 proc_lsst.multi:170 [INFO] [multi] job_id 17010049 proc_lsst.multi INFO: [multi] job_id 17010049 2024-03-13 09:45:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 09:45:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010049 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010049 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17010049 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-13 09:45:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-13 09:45:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 09:45:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 900 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/94ec93e4-5de8-4071-bcbe-d96c0c5c52c3_isr_898800_34.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/94ec93e4-5de8-4071-bcbe-d96c0c5c52c3_isr_898800_34.stderr parsl.dataflow.dflow DEBUG: Task 901 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 901 submitted for App characterizeImage, waiting on task 900 parsl.dataflow.dflow DEBUG: Task 901 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 901 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 902 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 902 submitted for App calibrate, waiting on task 901 parsl.dataflow.dflow DEBUG: Task 902 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 902 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 903 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 903 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 903 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 903 has memoization hash 98da60d14d823e98a29a6939fa90c077 parsl.dataflow.memoization INFO: Task 903 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c290180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 903 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/e690c0b5-b76b-4dba-b762-2d77bd62c06e_isr_898694_59.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/e690c0b5-b76b-4dba-b762-2d77bd62c06e_isr_898694_59.stderr parsl.dataflow.dflow DEBUG: Task 904 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 904 submitted for App characterizeImage, waiting on task 903 parsl.dataflow.dflow DEBUG: Task 904 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 904 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 905 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 905 submitted for App calibrate, waiting on task 904 parsl.dataflow.dflow DEBUG: Task 905 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 905 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 906 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 906 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 906 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 906 has memoization hash db57f933b01b5124e6c3369b9d50b544 parsl.dataflow.memoization INFO: Task 906 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2904a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 906 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/d15b739b-9b44-4c52-a722-1a77642503b8_isr_898696_3.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/d15b739b-9b44-4c52-a722-1a77642503b8_isr_898696_3.stderr parsl.dataflow.dflow DEBUG: Task 907 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 907 submitted for App characterizeImage, waiting on task 906 parsl.dataflow.dflow DEBUG: Task 907 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 907 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 908 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 908 submitted for App calibrate, waiting on task 907 parsl.dataflow.dflow DEBUG: Task 908 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 908 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 909 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 909 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 909 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 909 has memoization hash cd1b91e84a936d7533a3671a222e38ce parsl.dataflow.memoization INFO: Task 909 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2907c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 909 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/b5721661-4258-4291-b5a6-cd90379789ca_isr_898800_4.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/b5721661-4258-4291-b5a6-cd90379789ca_isr_898800_4.stderr parsl.dataflow.dflow DEBUG: Task 910 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 910 submitted for App characterizeImage, waiting on task 909 parsl.dataflow.dflow DEBUG: Task 910 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 910 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 911 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 911 submitted for App calibrate, waiting on task 910 parsl.dataflow.dflow DEBUG: Task 911 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 911 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 912 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 912 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 912 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 912 has memoization hash 2ce011445cfa4569b2e0dd62cbf15874 parsl.dataflow.memoization INFO: Task 912 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c290ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 912 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/01802909-7f15-4e3d-99d4-3d284d3c149c_isr_898799_12.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/01802909-7f15-4e3d-99d4-3d284d3c149c_isr_898799_12.stderr parsl.dataflow.dflow DEBUG: Task 913 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 913 submitted for App characterizeImage, waiting on task 912 parsl.dataflow.dflow DEBUG: Task 913 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 913 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 914 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 914 submitted for App calibrate, waiting on task 913 parsl.dataflow.dflow DEBUG: Task 914 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 914 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 915 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 915 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 915 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 915 has memoization hash 07ae30a681c21c17e9cac492677ae7a6 parsl.dataflow.memoization INFO: Task 915 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c290e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 915 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/27dde754-135c-4b11-b245-0385e7122dfa_isr_898800_14.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/27dde754-135c-4b11-b245-0385e7122dfa_isr_898800_14.stderr parsl.dataflow.dflow DEBUG: Task 916 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 916 submitted for App characterizeImage, waiting on task 915 parsl.dataflow.dflow DEBUG: Task 916 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 916 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 917 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 917 submitted for App calibrate, waiting on task 916 parsl.dataflow.dflow DEBUG: Task 917 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 917 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 918 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 918 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 918 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 918 has memoization hash efa7adff388e866ead39b6432ee70edf parsl.dataflow.memoization INFO: Task 918 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c291120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 918 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/e2814dbe-7be7-49e0-8382-3d36c39f8f57_isr_898800_5.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/e2814dbe-7be7-49e0-8382-3d36c39f8f57_isr_898800_5.stderr parsl.dataflow.dflow DEBUG: Task 919 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 919 submitted for App characterizeImage, waiting on task 918 parsl.dataflow.dflow DEBUG: Task 919 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 919 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 920 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 920 submitted for App calibrate, waiting on task 919 parsl.dataflow.dflow DEBUG: Task 920 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 920 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 921 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 921 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 921 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 921 has memoization hash 9e0844a347f35e64fdd56ed5792b524b parsl.dataflow.memoization INFO: Task 921 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c293d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 921 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/13930b43-43a3-42be-bbe5-f1010b32489e_isr_898695_13.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/13930b43-43a3-42be-bbe5-f1010b32489e_isr_898695_13.stderr parsl.dataflow.dflow DEBUG: Task 922 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 922 submitted for App characterizeImage, waiting on task 921 parsl.dataflow.dflow DEBUG: Task 922 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 922 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 923 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 923 submitted for App calibrate, waiting on task 922 parsl.dataflow.dflow DEBUG: Task 923 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 923 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 924 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 924 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 924 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 924 has memoization hash 6a7ea0884b7fb0552b9e537c0da293bb parsl.dataflow.memoization INFO: Task 924 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c293a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 924 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/49a1d469-b30e-46f9-a168-b862ce51b6c3_isr_898695_50.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/49a1d469-b30e-46f9-a168-b862ce51b6c3_isr_898695_50.stderr parsl.dataflow.dflow DEBUG: Task 925 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 925 submitted for App characterizeImage, waiting on task 924 parsl.dataflow.dflow DEBUG: Task 925 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 925 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 926 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 926 submitted for App calibrate, waiting on task 925 parsl.dataflow.dflow DEBUG: Task 926 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 926 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 927 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 927 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 927 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 927 has memoization hash 642b1d07959ed926cfaa1f4ce8195965 parsl.dataflow.memoization INFO: Task 927 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c293740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 927 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/3056ed80-d157-44a2-9e32-6edd6be6f185_isr_898696_16.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/3056ed80-d157-44a2-9e32-6edd6be6f185_isr_898696_16.stderr parsl.dataflow.dflow DEBUG: Task 928 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 928 submitted for App characterizeImage, waiting on task 927 parsl.dataflow.dflow DEBUG: Task 928 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 928 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 929 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 929 submitted for App calibrate, waiting on task 928 parsl.dataflow.dflow DEBUG: Task 929 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 929 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 930 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 930 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 930 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 930 has memoization hash ed71ef4d5295ef94c1fcd04721b12c99 parsl.dataflow.memoization INFO: Task 930 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2916c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 930 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/415cbfb9-bfd1-4903-a24e-71da1df10682_isr_898694_9.stdout parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/415cbfb9-bfd1-4903-a24e-71da1df10682_isr_898694_9.stderr parsl.dataflow.dflow DEBUG: Task 931 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 931 submitted for App characterizeImage, waiting on task 930 parsl.dataflow.dflow DEBUG: Task 931 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 931 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 932 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 932 submitted for App calibrate, waiting on task 931 parsl.dataflow.dflow DEBUG: Task 932 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 932 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 933 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 933 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 933 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 933 has memoization hash cef45b7226cf882690f7fc055003b34f parsl.dataflow.memoization INFO: Task 933 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c291940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 933 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/9e7d918f-917f-40e8-8e47-be7a18be6f45_isr_898696_33.stdout parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/9e7d918f-917f-40e8-8e47-be7a18be6f45_isr_898696_33.stderr parsl.dataflow.dflow DEBUG: Task 934 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 934 submitted for App characterizeImage, waiting on task 933 parsl.dataflow.dflow DEBUG: Task 934 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 934 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 935 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 935 submitted for App calibrate, waiting on task 934 parsl.dataflow.dflow DEBUG: Task 935 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 935 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 936 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 936 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 936 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 936 has memoization hash 4ed308d914577fab3fc6680ad3f34a67 parsl.dataflow.memoization INFO: Task 936 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c291c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 936 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/80751463-1bac-4b6e-8d93-f844ac5a7b23_isr_898695_28.stdout parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/80751463-1bac-4b6e-8d93-f844ac5a7b23_isr_898695_28.stderr parsl.dataflow.dflow DEBUG: Task 937 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 937 submitted for App characterizeImage, waiting on task 936 parsl.dataflow.dflow DEBUG: Task 937 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 937 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 938 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 938 submitted for App calibrate, waiting on task 937 parsl.dataflow.dflow DEBUG: Task 938 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 938 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 939 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 939 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 939 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 939 has memoization hash 1e35c7dc1fbb6c8d5e43011778c55902 parsl.dataflow.memoization INFO: Task 939 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c291da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 939 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/f87600ed-2b18-487f-ae2e-765a70c10a81_isr_898798_21.stdout parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/f87600ed-2b18-487f-ae2e-765a70c10a81_isr_898798_21.stderr parsl.dataflow.dflow DEBUG: Task 940 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 940 submitted for App characterizeImage, waiting on task 939 parsl.dataflow.dflow DEBUG: Task 940 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 940 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 941 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 941 submitted for App calibrate, waiting on task 940 parsl.dataflow.dflow DEBUG: Task 941 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 941 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 942 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 942 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 942 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 942 has memoization hash f63c7a0e33c34f2564691649cbee7e7a parsl.dataflow.memoization INFO: Task 942 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2920c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 942 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/c12239ed-1b55-421a-b23b-3e2d4aa3fdb8_isr_898800_61.stdout parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/c12239ed-1b55-421a-b23b-3e2d4aa3fdb8_isr_898800_61.stderr parsl.dataflow.dflow DEBUG: Task 943 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 943 submitted for App characterizeImage, waiting on task 942 parsl.dataflow.dflow DEBUG: Task 943 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 943 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 944 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 944 submitted for App calibrate, waiting on task 943 parsl.dataflow.dflow DEBUG: Task 944 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 944 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 945 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 945 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 945 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 945 has memoization hash 298f60f989a0cd803bf0dfbbf3a87b19 parsl.dataflow.memoization INFO: Task 945 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c292520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 945 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/1f43662c-6043-4ed6-b3b9-b58556c5c804_isr_898799_9.stdout parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/1f43662c-6043-4ed6-b3b9-b58556c5c804_isr_898799_9.stderr parsl.dataflow.dflow DEBUG: Task 946 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 946 submitted for App characterizeImage, waiting on task 945 parsl.dataflow.dflow DEBUG: Task 946 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 946 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 947 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 947 submitted for App calibrate, waiting on task 946 parsl.dataflow.dflow DEBUG: Task 947 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 947 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 948 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 948 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 948 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 948 has memoization hash 91681ccd679d5700f7468c879400c7ed parsl.dataflow.memoization INFO: Task 948 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c292840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 948 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/f5ae34b4-acc7-441a-8121-e300214535b0_isr_898800_2.stdout parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/f5ae34b4-acc7-441a-8121-e300214535b0_isr_898800_2.stderr parsl.dataflow.dflow DEBUG: Task 949 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 949 submitted for App characterizeImage, waiting on task 948 parsl.dataflow.dflow DEBUG: Task 949 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 949 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 950 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 950 submitted for App calibrate, waiting on task 949 parsl.dataflow.dflow DEBUG: Task 950 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 950 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 951 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 951 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 951 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 951 has memoization hash ba85e32263ccd133272544a489043059 parsl.dataflow.memoization INFO: Task 951 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c292ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 951 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/89a19fed-e26a-4ef9-b449-03cd45c40ed8_isr_898798_16.stdout parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/89a19fed-e26a-4ef9-b449-03cd45c40ed8_isr_898798_16.stderr parsl.dataflow.dflow DEBUG: Task 952 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 952 submitted for App characterizeImage, waiting on task 951 parsl.dataflow.dflow DEBUG: Task 952 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 952 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 953 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 953 submitted for App calibrate, waiting on task 952 parsl.dataflow.dflow DEBUG: Task 953 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 953 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 954 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 954 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 954 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 954 has memoization hash 69dd30147106005772c929630b1f8556 parsl.dataflow.memoization INFO: Task 954 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c292fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 954 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/24352de4-a006-4174-a5b2-f0c6ae7d5323_isr_898798_1.stdout parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/24352de4-a006-4174-a5b2-f0c6ae7d5323_isr_898798_1.stderr parsl.dataflow.dflow DEBUG: Task 955 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 955 submitted for App characterizeImage, waiting on task 954 parsl.dataflow.dflow DEBUG: Task 955 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 955 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 956 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 956 submitted for App calibrate, waiting on task 955 parsl.dataflow.dflow DEBUG: Task 956 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 956 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 957 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 957 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 957 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 957 has memoization hash 2d45d63147ca03f42e51be96057330f9 parsl.dataflow.memoization INFO: Task 957 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c293240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 957 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/942f86b5-b8b7-423d-aeaf-7154bea804c1_isr_898798_51.stdout parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/942f86b5-b8b7-423d-aeaf-7154bea804c1_isr_898798_51.stderr parsl.dataflow.dflow DEBUG: Task 958 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 958 submitted for App characterizeImage, waiting on task 957 parsl.dataflow.dflow DEBUG: Task 958 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 958 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 959 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 959 submitted for App calibrate, waiting on task 958 parsl.dataflow.dflow DEBUG: Task 959 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 959 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 960 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 960 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 960 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 960 has memoization hash d9036686abcf3d14dc46ae9ebc1fa5b9 parsl.dataflow.memoization INFO: Task 960 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c293420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 960 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/528f690a-7c33-479c-a3a9-8567587011eb_isr_898800_3.stdout parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/528f690a-7c33-479c-a3a9-8567587011eb_isr_898800_3.stderr parsl.dataflow.dflow DEBUG: Task 961 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 961 submitted for App characterizeImage, waiting on task 960 parsl.dataflow.dflow DEBUG: Task 961 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 961 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 962 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 962 submitted for App calibrate, waiting on task 961 parsl.dataflow.dflow DEBUG: Task 962 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 962 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 963 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 963 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 963 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 963 has memoization hash 57fac441bc04f526214fd85b25384537 parsl.dataflow.memoization INFO: Task 963 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 963 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/4ea13f91-3392-4436-8ac4-0255d857cd23_isr_898799_51.stdout parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/4ea13f91-3392-4436-8ac4-0255d857cd23_isr_898799_51.stderr parsl.dataflow.dflow DEBUG: Task 964 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 964 submitted for App characterizeImage, waiting on task 963 parsl.dataflow.dflow DEBUG: Task 964 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 964 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 965 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 965 submitted for App calibrate, waiting on task 964 parsl.dataflow.dflow DEBUG: Task 965 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 965 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 966 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 966 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 966 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 966 has memoization hash 0708c81c9e0d7a6a74c2d5804a259f22 parsl.dataflow.memoization INFO: Task 966 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 966 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/a116b3e4-de6d-4fc9-87aa-0fad11b6987b_isr_898695_36.stdout parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/a116b3e4-de6d-4fc9-87aa-0fad11b6987b_isr_898695_36.stderr parsl.dataflow.dflow DEBUG: Task 967 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 967 submitted for App characterizeImage, waiting on task 966 parsl.dataflow.dflow DEBUG: Task 967 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 967 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 968 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 968 submitted for App calibrate, waiting on task 967 parsl.dataflow.dflow DEBUG: Task 968 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 968 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 969 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 969 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 969 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 969 has memoization hash e05afe3500ad3e46b6a4b9a00e8e45d9 parsl.dataflow.memoization INFO: Task 969 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 969 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/cb0ea134-aafd-498d-8f49-eb21e2717a50_isr_898695_20.stdout parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/cb0ea134-aafd-498d-8f49-eb21e2717a50_isr_898695_20.stderr parsl.dataflow.dflow DEBUG: Task 970 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 970 submitted for App characterizeImage, waiting on task 969 parsl.dataflow.dflow DEBUG: Task 970 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 970 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 971 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 971 submitted for App calibrate, waiting on task 970 parsl.dataflow.dflow DEBUG: Task 971 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 971 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 972 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 972 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 972 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 972 has memoization hash b275d64ec2b91b6a13b93c23326f1ec0 parsl.dataflow.memoization INFO: Task 972 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 972 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/589ace7e-a037-4dcb-9c8f-aee3432de889_isr_898696_15.stdout parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/589ace7e-a037-4dcb-9c8f-aee3432de889_isr_898696_15.stderr parsl.dataflow.dflow DEBUG: Task 973 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 973 submitted for App characterizeImage, waiting on task 972 parsl.dataflow.dflow DEBUG: Task 973 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 973 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 974 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 974 submitted for App calibrate, waiting on task 973 parsl.dataflow.dflow DEBUG: Task 974 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 974 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 975 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 975 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 975 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 975 has memoization hash 85a1d611edba099b8eee9165d0b2f417 parsl.dataflow.memoization INFO: Task 975 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 975 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/a50169d3-940b-4f30-bf83-7eac842ddd58_isr_898799_57.stdout parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/a50169d3-940b-4f30-bf83-7eac842ddd58_isr_898799_57.stderr parsl.dataflow.dflow DEBUG: Task 976 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 976 submitted for App characterizeImage, waiting on task 975 parsl.dataflow.dflow DEBUG: Task 976 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 976 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 977 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 977 submitted for App calibrate, waiting on task 976 parsl.dataflow.dflow DEBUG: Task 977 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 977 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 978 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 978 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 978 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 978 has memoization hash 331f5206bc5623b482504a0c6fc85427 parsl.dataflow.memoization INFO: Task 978 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 978 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/4ace23da-a838-4ce8-a91a-309b4a57afb2_isr_898800_21.stdout parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/4ace23da-a838-4ce8-a91a-309b4a57afb2_isr_898800_21.stderr parsl.dataflow.dflow DEBUG: Task 979 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 979 submitted for App characterizeImage, waiting on task 978 parsl.dataflow.dflow DEBUG: Task 979 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 979 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 980 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 980 submitted for App calibrate, waiting on task 979 parsl.dataflow.dflow DEBUG: Task 980 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 980 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 981 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 981 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 981 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 981 has memoization hash 1061638247b5e51ad7831b6a07931544 parsl.dataflow.memoization INFO: Task 981 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 981 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/c1796cee-3051-4d87-889b-7692d5d7ea9f_isr_898696_34.stdout parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/c1796cee-3051-4d87-889b-7692d5d7ea9f_isr_898696_34.stderr parsl.dataflow.dflow DEBUG: Task 982 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 982 submitted for App characterizeImage, waiting on task 981 parsl.dataflow.dflow DEBUG: Task 982 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 982 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 983 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 983 submitted for App calibrate, waiting on task 982 parsl.dataflow.dflow DEBUG: Task 983 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 983 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 984 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 984 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 984 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 984 has memoization hash 8be52af19f518a72d39b5fe27c98fe70 parsl.dataflow.memoization INFO: Task 984 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 984 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/28db2c3d-f2a4-49db-9793-9f3b7eda582b_isr_898798_12.stdout parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/28db2c3d-f2a4-49db-9793-9f3b7eda582b_isr_898798_12.stderr parsl.dataflow.dflow DEBUG: Task 985 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 985 submitted for App characterizeImage, waiting on task 984 parsl.dataflow.dflow DEBUG: Task 985 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 985 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 986 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 986 submitted for App calibrate, waiting on task 985 parsl.dataflow.dflow DEBUG: Task 986 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 986 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 987 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 987 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 987 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 987 has memoization hash 9dfcd2138dbfa0b654eb5e09737d82af parsl.dataflow.memoization INFO: Task 987 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 987 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/5a923c84-5011-4e29-b3a7-54626f2345ae_isr_898695_61.stdout parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/5a923c84-5011-4e29-b3a7-54626f2345ae_isr_898695_61.stderr parsl.dataflow.dflow DEBUG: Task 988 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 988 submitted for App characterizeImage, waiting on task 987 parsl.dataflow.dflow DEBUG: Task 988 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 988 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 989 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 989 submitted for App calibrate, waiting on task 988 parsl.dataflow.dflow DEBUG: Task 989 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 989 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 990 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 990 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 990 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 990 has memoization hash 288a89d2c2129d6baf1063e9b14df990 parsl.dataflow.memoization INFO: Task 990 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 990 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/09a3c603-27d1-4ae5-84d2-ecf4610c67e5_isr_898694_61.stdout parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/09a3c603-27d1-4ae5-84d2-ecf4610c67e5_isr_898694_61.stderr parsl.dataflow.dflow DEBUG: Task 991 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 991 submitted for App characterizeImage, waiting on task 990 parsl.dataflow.dflow DEBUG: Task 991 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 991 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 992 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 992 submitted for App calibrate, waiting on task 991 parsl.dataflow.dflow DEBUG: Task 992 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 992 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 993 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 993 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 993 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 993 has memoization hash f35b9a565c0b6c36f803a1d309e6a55f parsl.dataflow.memoization INFO: Task 993 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2da160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 993 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/bcd85c87-3023-428f-accd-32fc43f39a67_isr_898798_7.stdout parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/bcd85c87-3023-428f-accd-32fc43f39a67_isr_898798_7.stderr parsl.dataflow.dflow DEBUG: Task 994 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 994 submitted for App characterizeImage, waiting on task 993 parsl.dataflow.dflow DEBUG: Task 994 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 994 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 995 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 995 submitted for App calibrate, waiting on task 994 parsl.dataflow.dflow DEBUG: Task 995 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 995 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 996 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 996 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 996 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 996 has memoization hash fdb02983a0e0bd09c9b0a4faeb030690 parsl.dataflow.memoization INFO: Task 996 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2da480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 996 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/34e14fc7-c24b-4ab4-a7e7-30fc1824f167_isr_898798_19.stdout parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/34e14fc7-c24b-4ab4-a7e7-30fc1824f167_isr_898798_19.stderr parsl.dataflow.dflow DEBUG: Task 997 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 997 submitted for App characterizeImage, waiting on task 996 parsl.dataflow.dflow DEBUG: Task 997 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 997 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 998 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 998 submitted for App calibrate, waiting on task 997 parsl.dataflow.dflow DEBUG: Task 998 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 998 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 999 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 999 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 999 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 999 has memoization hash 0a370f0b0b1a480926948e92b00298ea parsl.dataflow.memoization INFO: Task 999 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2da7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 999 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/b05f1916-e33d-4ce4-ab19-02f5cfcfafcf_isr_898695_42.stdout parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/b05f1916-e33d-4ce4-ab19-02f5cfcfafcf_isr_898695_42.stderr parsl.dataflow.dflow DEBUG: Task 1000 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1000 submitted for App characterizeImage, waiting on task 999 parsl.dataflow.dflow DEBUG: Task 1000 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1000 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1001 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1001 submitted for App calibrate, waiting on task 1000 parsl.dataflow.dflow DEBUG: Task 1001 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1001 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1002 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1002 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1002 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1002 has memoization hash 7374066c9cc9d3b9a2081043001493c2 parsl.dataflow.memoization INFO: Task 1002 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2daac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1002 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/570da354-59d2-4e18-a705-9d622317b64c_isr_898799_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/570da354-59d2-4e18-a705-9d622317b64c_isr_898799_53.stderr parsl.dataflow.dflow DEBUG: Task 1003 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1003 submitted for App characterizeImage, waiting on task 1002 parsl.dataflow.dflow DEBUG: Task 1003 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1003 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1004 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1004 submitted for App calibrate, waiting on task 1003 parsl.dataflow.dflow DEBUG: Task 1004 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1004 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1005 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1005 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1005 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1005 has memoization hash fbf6c7e557f9f79e8d7270c49c75e463 parsl.dataflow.memoization INFO: Task 1005 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2dade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1005 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/317817ee-e8d5-4c40-8de7-a5445f2f6d1a_isr_898695_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/317817ee-e8d5-4c40-8de7-a5445f2f6d1a_isr_898695_38.stderr parsl.dataflow.dflow DEBUG: Task 1006 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1006 submitted for App characterizeImage, waiting on task 1005 parsl.dataflow.dflow DEBUG: Task 1006 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1006 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1007 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1007 submitted for App calibrate, waiting on task 1006 parsl.dataflow.dflow DEBUG: Task 1007 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1007 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1008 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1008 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1008 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1008 has memoization hash 599d1a91e2458b4e96cf0345e512b3ad parsl.dataflow.memoization INFO: Task 1008 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2db100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1008 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/8f81b55f-fc28-47a4-bbe7-38d5d32781e9_isr_898798_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/8f81b55f-fc28-47a4-bbe7-38d5d32781e9_isr_898798_8.stderr parsl.dataflow.dflow DEBUG: Task 1009 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1009 submitted for App characterizeImage, waiting on task 1008 parsl.dataflow.dflow DEBUG: Task 1009 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1009 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1010 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1010 submitted for App calibrate, waiting on task 1009 parsl.dataflow.dflow DEBUG: Task 1010 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1010 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1011 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1011 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1011 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1011 has memoization hash a9f8e964598696b7812414e242a0fd4e parsl.dataflow.memoization INFO: Task 1011 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2db420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1011 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/d7c86482-ed31-462f-93d2-f623b2c14586_isr_898696_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/d7c86482-ed31-462f-93d2-f623b2c14586_isr_898696_24.stderr parsl.dataflow.dflow DEBUG: Task 1012 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1012 submitted for App characterizeImage, waiting on task 1011 parsl.dataflow.dflow DEBUG: Task 1012 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1012 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1013 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1013 submitted for App calibrate, waiting on task 1012 parsl.dataflow.dflow DEBUG: Task 1013 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1013 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1014 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1014 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1014 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1014 has memoization hash 84b70a9d19b71adfb3a98be1775af819 parsl.dataflow.memoization INFO: Task 1014 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2db740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 09:45:10 proc_lsst.multi:170 [INFO] [multi] job_id 17010050 proc_lsst.multi INFO: [multi] job_id 17010050 2024-03-13 09:45:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010050 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010050 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17010050 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-13 09:45:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-13 09:45:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1014 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/dbd3525a-55f0-494e-a27d-a353be4bc48f_isr_898696_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/dbd3525a-55f0-494e-a27d-a353be4bc48f_isr_898696_45.stderr parsl.dataflow.dflow DEBUG: Task 1015 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1015 submitted for App characterizeImage, waiting on task 1014 parsl.dataflow.dflow DEBUG: Task 1015 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1015 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1016 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1016 submitted for App calibrate, waiting on task 1015 parsl.dataflow.dflow DEBUG: Task 1016 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1016 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1017 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1017 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1017 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1017 has memoization hash 8abd7631dc389cb69c03b631d389fee1 parsl.dataflow.memoization INFO: Task 1017 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2db7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1017 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/3999cc0b-404a-4fc3-b8fa-a0e2fe5c3911_isr_898800_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/3999cc0b-404a-4fc3-b8fa-a0e2fe5c3911_isr_898800_22.stderr parsl.dataflow.dflow DEBUG: Task 1018 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1018 submitted for App characterizeImage, waiting on task 1017 parsl.dataflow.dflow DEBUG: Task 1018 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1018 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1019 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1019 submitted for App calibrate, waiting on task 1018 parsl.dataflow.dflow DEBUG: Task 1019 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1019 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1020 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1020 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1020 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1020 has memoization hash fb3e89ce038244af0773794dbc3ccccd parsl.dataflow.memoization INFO: Task 1020 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2dbba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1020 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/64c91e49-24df-4b19-95f0-e2ceffae365b_isr_898695_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/64c91e49-24df-4b19-95f0-e2ceffae365b_isr_898695_33.stderr parsl.dataflow.dflow DEBUG: Task 1021 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1021 submitted for App characterizeImage, waiting on task 1020 parsl.dataflow.dflow DEBUG: Task 1021 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1021 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1022 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1022 submitted for App calibrate, waiting on task 1021 parsl.dataflow.dflow DEBUG: Task 1022 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1022 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1023 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1023 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1023 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1023 has memoization hash b4047848b1ac29dd3391f3f4e94b50fc parsl.dataflow.memoization INFO: Task 1023 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2dbec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1023 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/39fb7fe2-7378-4f67-a238-cd4404b9ebfc_isr_898800_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/39fb7fe2-7378-4f67-a238-cd4404b9ebfc_isr_898800_6.stderr parsl.dataflow.dflow DEBUG: Task 1024 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1024 submitted for App characterizeImage, waiting on task 1023 parsl.dataflow.dflow DEBUG: Task 1024 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1024 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1025 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1025 submitted for App calibrate, waiting on task 1024 parsl.dataflow.dflow DEBUG: Task 1025 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1025 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1026 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1026 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1026 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1026 has memoization hash 04695d397fcdc5dba321a3859d2d700a parsl.dataflow.memoization INFO: Task 1026 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c328220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1026 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/3d840236-f1ba-48f8-a3d1-e314b85e477e_isr_898694_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/3d840236-f1ba-48f8-a3d1-e314b85e477e_isr_898694_13.stderr parsl.dataflow.dflow DEBUG: Task 1027 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1027 submitted for App characterizeImage, waiting on task 1026 parsl.dataflow.dflow DEBUG: Task 1027 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1027 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1028 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1028 submitted for App calibrate, waiting on task 1027 parsl.dataflow.dflow DEBUG: Task 1028 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1028 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1029 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1029 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1029 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1029 has memoization hash f9fab1858216aeb7a49a77d9c8567249 parsl.dataflow.memoization INFO: Task 1029 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1029 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/09646f2b-0617-4d05-9dc5-c71fd0716f24_isr_898798_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/09646f2b-0617-4d05-9dc5-c71fd0716f24_isr_898798_14.stderr parsl.dataflow.dflow DEBUG: Task 1030 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1030 submitted for App characterizeImage, waiting on task 1029 parsl.dataflow.dflow DEBUG: Task 1030 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1030 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1031 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1031 submitted for App calibrate, waiting on task 1030 parsl.dataflow.dflow DEBUG: Task 1031 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1031 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1032 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1032 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1032 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1032 has memoization hash eeb49c5f1bf4fa7baabb81f5ce555aee parsl.dataflow.memoization INFO: Task 1032 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1032 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/6d6dc98b-2ec8-40eb-afb9-c6fbc637c7dd_isr_898798_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/6d6dc98b-2ec8-40eb-afb9-c6fbc637c7dd_isr_898798_26.stderr parsl.dataflow.dflow DEBUG: Task 1033 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1033 submitted for App characterizeImage, waiting on task 1032 parsl.dataflow.dflow DEBUG: Task 1033 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1033 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1034 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1034 submitted for App calibrate, waiting on task 1033 parsl.dataflow.dflow DEBUG: Task 1034 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1034 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1035 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1035 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1035 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1035 has memoization hash 8350707472b0085c99611d08363fbda2 parsl.dataflow.memoization INFO: Task 1035 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1035 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/880f63c9-3547-4b22-a543-c72da297c625_isr_898694_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/880f63c9-3547-4b22-a543-c72da297c625_isr_898694_33.stderr parsl.dataflow.dflow DEBUG: Task 1036 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1036 submitted for App characterizeImage, waiting on task 1035 parsl.dataflow.dflow DEBUG: Task 1036 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1036 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1037 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1037 submitted for App calibrate, waiting on task 1036 parsl.dataflow.dflow DEBUG: Task 1037 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1037 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1038 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1038 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1038 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1038 has memoization hash 739cbd7a5d58e2596a954c162b86f13d parsl.dataflow.memoization INFO: Task 1038 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1038 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/95efd55a-6361-468b-a475-d41d582d9ade_isr_898800_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/95efd55a-6361-468b-a475-d41d582d9ade_isr_898800_33.stderr parsl.dataflow.dflow DEBUG: Task 1039 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1039 submitted for App characterizeImage, waiting on task 1038 parsl.dataflow.dflow DEBUG: Task 1039 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1039 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1040 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1040 submitted for App calibrate, waiting on task 1039 parsl.dataflow.dflow DEBUG: Task 1040 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1040 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1041 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1041 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1041 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1041 has memoization hash 65a807dfc99d8848e4abc63691577e71 parsl.dataflow.memoization INFO: Task 1041 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1041 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/167e2666-4b68-4b1a-8f71-85edc21cb6ab_isr_898798_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/167e2666-4b68-4b1a-8f71-85edc21cb6ab_isr_898798_27.stderr parsl.dataflow.dflow DEBUG: Task 1042 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1042 submitted for App characterizeImage, waiting on task 1041 parsl.dataflow.dflow DEBUG: Task 1042 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1042 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1043 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1043 submitted for App calibrate, waiting on task 1042 parsl.dataflow.dflow DEBUG: Task 1043 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1043 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1044 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1044 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1044 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1044 has memoization hash df41f479eeb49a7364227c822ab92d6f parsl.dataflow.memoization INFO: Task 1044 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1044 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/96c5124e-15ad-4874-8036-7e4f67ec23c6_isr_898800_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/96c5124e-15ad-4874-8036-7e4f67ec23c6_isr_898800_8.stderr parsl.dataflow.dflow DEBUG: Task 1045 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1045 submitted for App characterizeImage, waiting on task 1044 parsl.dataflow.dflow DEBUG: Task 1045 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1045 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1046 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1046 submitted for App calibrate, waiting on task 1045 parsl.dataflow.dflow DEBUG: Task 1046 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1046 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1047 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1047 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1047 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1047 has memoization hash ed7af9acd016eb5855892b8f18440ff0 parsl.dataflow.memoization INFO: Task 1047 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1047 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/98986982-3857-4ae7-9f8c-e2129b5b182e_isr_898696_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/98986982-3857-4ae7-9f8c-e2129b5b182e_isr_898696_9.stderr parsl.dataflow.dflow DEBUG: Task 1048 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1048 submitted for App characterizeImage, waiting on task 1047 parsl.dataflow.dflow DEBUG: Task 1048 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1048 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1049 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1049 submitted for App calibrate, waiting on task 1048 parsl.dataflow.dflow DEBUG: Task 1049 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1049 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1050 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1050 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1050 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1050 has memoization hash f467a3d1a5a25d6427e2a83691ea7047 parsl.dataflow.memoization INFO: Task 1050 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1050 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/d4fc5835-d031-4b21-a3c0-c800b7503e61_isr_898800_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/d4fc5835-d031-4b21-a3c0-c800b7503e61_isr_898800_9.stderr parsl.dataflow.dflow DEBUG: Task 1051 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1051 submitted for App characterizeImage, waiting on task 1050 parsl.dataflow.dflow DEBUG: Task 1051 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1051 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1052 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1052 submitted for App calibrate, waiting on task 1051 parsl.dataflow.dflow DEBUG: Task 1052 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1052 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1053 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1053 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1053 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1053 has memoization hash d5244404b80bea21db020634059c7054 parsl.dataflow.memoization INFO: Task 1053 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1053 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/3399a077-2074-4bf4-a72a-9c1dcba947f8_isr_898800_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/3399a077-2074-4bf4-a72a-9c1dcba947f8_isr_898800_55.stderr parsl.dataflow.dflow DEBUG: Task 1054 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1054 submitted for App characterizeImage, waiting on task 1053 parsl.dataflow.dflow DEBUG: Task 1054 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1054 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1055 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1055 submitted for App calibrate, waiting on task 1054 parsl.dataflow.dflow DEBUG: Task 1055 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1055 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1056 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1056 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1056 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1056 has memoization hash 02219285c93082f33cc930c29d28e664 parsl.dataflow.memoization INFO: Task 1056 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3502c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1056 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/20a5a27c-f9c9-4f18-83f9-1dfcb18b06df_isr_898798_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/20a5a27c-f9c9-4f18-83f9-1dfcb18b06df_isr_898798_48.stderr parsl.dataflow.dflow DEBUG: Task 1057 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1057 submitted for App characterizeImage, waiting on task 1056 parsl.dataflow.dflow DEBUG: Task 1057 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1057 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1058 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1058 submitted for App calibrate, waiting on task 1057 parsl.dataflow.dflow DEBUG: Task 1058 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1058 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1059 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1059 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1059 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1059 has memoization hash dee01d0ef34baadc219adf0b4067a734 parsl.dataflow.memoization INFO: Task 1059 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c328900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1059 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/ec724237-1bd6-4e96-aa1e-87f376aa0abb_isr_898696_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/ec724237-1bd6-4e96-aa1e-87f376aa0abb_isr_898696_61.stderr parsl.dataflow.dflow DEBUG: Task 1060 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1060 submitted for App characterizeImage, waiting on task 1059 parsl.dataflow.dflow DEBUG: Task 1060 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1060 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1061 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1061 submitted for App calibrate, waiting on task 1060 parsl.dataflow.dflow DEBUG: Task 1061 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1061 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1062 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1062 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1062 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1062 has memoization hash f2feb32e37bca754ac4cfb23b57a45a8 parsl.dataflow.memoization INFO: Task 1062 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c328cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1062 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/267a88bd-3492-496c-b04d-cb88f5756c3d_isr_898695_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/267a88bd-3492-496c-b04d-cb88f5756c3d_isr_898695_9.stderr parsl.dataflow.dflow DEBUG: Task 1063 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1063 submitted for App characterizeImage, waiting on task 1062 parsl.dataflow.dflow DEBUG: Task 1063 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1063 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1064 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1064 submitted for App calibrate, waiting on task 1063 parsl.dataflow.dflow DEBUG: Task 1064 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1064 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1065 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1065 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1065 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1065 has memoization hash 02cf6bb00c5b995d36b324f013dd5ffa parsl.dataflow.memoization INFO: Task 1065 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c329080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1065 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/05f2e988-e934-4a41-996e-225f96d01d9f_isr_898798_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/05f2e988-e934-4a41-996e-225f96d01d9f_isr_898798_37.stderr parsl.dataflow.dflow DEBUG: Task 1066 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1066 submitted for App characterizeImage, waiting on task 1065 parsl.dataflow.dflow DEBUG: Task 1066 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1066 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1067 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1067 submitted for App calibrate, waiting on task 1066 parsl.dataflow.dflow DEBUG: Task 1067 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1067 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1068 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1068 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1068 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1068 has memoization hash 827ca6d402c80937d49dd2bee32d6898 parsl.dataflow.memoization INFO: Task 1068 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3293a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1068 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/202eb645-60e1-4530-99f0-987ec51af627_isr_898696_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/202eb645-60e1-4530-99f0-987ec51af627_isr_898696_11.stderr parsl.dataflow.dflow DEBUG: Task 1069 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1069 submitted for App characterizeImage, waiting on task 1068 parsl.dataflow.dflow DEBUG: Task 1069 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1069 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1070 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1070 submitted for App calibrate, waiting on task 1069 parsl.dataflow.dflow DEBUG: Task 1070 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1070 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1071 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1071 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1071 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1071 has memoization hash 94cc2cc4dae02319eb90d6e82da2bce4 parsl.dataflow.memoization INFO: Task 1071 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3296c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1071 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/a9794ef2-5a2b-4a12-8f81-3bee8acebef6_isr_898799_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/a9794ef2-5a2b-4a12-8f81-3bee8acebef6_isr_898799_43.stderr parsl.dataflow.dflow DEBUG: Task 1072 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1072 submitted for App characterizeImage, waiting on task 1071 parsl.dataflow.dflow DEBUG: Task 1072 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1072 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1073 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1073 submitted for App calibrate, waiting on task 1072 parsl.dataflow.dflow DEBUG: Task 1073 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1073 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1074 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1074 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1074 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1074 has memoization hash b521dd290c498a77b87467ce2133bd45 parsl.dataflow.memoization INFO: Task 1074 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3299e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1074 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/4f1ce603-c88f-400b-a4bf-fcaf9cbabf46_isr_898695_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/4f1ce603-c88f-400b-a4bf-fcaf9cbabf46_isr_898695_15.stderr parsl.dataflow.dflow DEBUG: Task 1075 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1075 submitted for App characterizeImage, waiting on task 1074 parsl.dataflow.dflow DEBUG: Task 1075 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1075 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1076 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1076 submitted for App calibrate, waiting on task 1075 parsl.dataflow.dflow DEBUG: Task 1076 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1076 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1077 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1077 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1077 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1077 has memoization hash d877b16c2bed5a7ace5089d2665433c9 parsl.dataflow.memoization INFO: Task 1077 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c329d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1077 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/c25bfb1a-851b-43fd-a73f-d543ae8c97a7_isr_898696_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/c25bfb1a-851b-43fd-a73f-d543ae8c97a7_isr_898696_58.stderr parsl.dataflow.dflow DEBUG: Task 1078 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1078 submitted for App characterizeImage, waiting on task 1077 parsl.dataflow.dflow DEBUG: Task 1078 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1078 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1079 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1079 submitted for App calibrate, waiting on task 1078 parsl.dataflow.dflow DEBUG: Task 1079 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1079 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1080 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1080 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1080 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1080 has memoization hash 557ed074d027670f715cf4a1601bac2c parsl.dataflow.memoization INFO: Task 1080 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c329da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1080 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/3026279c-3fb0-4e26-928d-fe9c223afe86_isr_898695_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/3026279c-3fb0-4e26-928d-fe9c223afe86_isr_898695_23.stderr parsl.dataflow.dflow DEBUG: Task 1081 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1081 submitted for App characterizeImage, waiting on task 1080 parsl.dataflow.dflow DEBUG: Task 1081 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1081 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1082 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1082 submitted for App calibrate, waiting on task 1081 parsl.dataflow.dflow DEBUG: Task 1082 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1082 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1083 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1083 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1083 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1083 has memoization hash 6cbf78a8569d3cd7531edc5c07eafad0 parsl.dataflow.memoization INFO: Task 1083 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1083 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/f2c015a5-f6c6-40b1-895b-1bf7538c7202_isr_898695_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/f2c015a5-f6c6-40b1-895b-1bf7538c7202_isr_898695_37.stderr parsl.dataflow.dflow DEBUG: Task 1084 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1084 submitted for App characterizeImage, waiting on task 1083 parsl.dataflow.dflow DEBUG: Task 1084 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1084 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1085 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1085 submitted for App calibrate, waiting on task 1084 parsl.dataflow.dflow DEBUG: Task 1085 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1085 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1086 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1086 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1086 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1086 has memoization hash aa47700a03f618dcf1fb2d255b0b0041 parsl.dataflow.memoization INFO: Task 1086 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c350180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1086 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/df0f69c1-5256-4757-807e-cfd544b104d9_isr_898800_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/df0f69c1-5256-4757-807e-cfd544b104d9_isr_898800_16.stderr parsl.dataflow.dflow DEBUG: Task 1087 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1087 submitted for App characterizeImage, waiting on task 1086 parsl.dataflow.dflow DEBUG: Task 1087 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1087 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1088 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1088 submitted for App calibrate, waiting on task 1087 parsl.dataflow.dflow DEBUG: Task 1088 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1088 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1089 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1089 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1089 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1089 has memoization hash 5efb92a40357fc83da13902075740a47 parsl.dataflow.memoization INFO: Task 1089 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3505e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1089 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/3d573d67-cfa2-4186-8a77-7282c927638b_isr_898696_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/3d573d67-cfa2-4186-8a77-7282c927638b_isr_898696_19.stderr parsl.dataflow.dflow DEBUG: Task 1090 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1090 submitted for App characterizeImage, waiting on task 1089 parsl.dataflow.dflow DEBUG: Task 1090 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1090 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1091 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1091 submitted for App calibrate, waiting on task 1090 parsl.dataflow.dflow DEBUG: Task 1091 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1091 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1092 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1092 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1092 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1092 has memoization hash 29c51cdc7d2d76dd6a063aea498dbc69 parsl.dataflow.memoization INFO: Task 1092 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c350900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1092 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/82decbe3-a1b8-40fc-ad05-ded92deea7cc_isr_898694_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/82decbe3-a1b8-40fc-ad05-ded92deea7cc_isr_898694_12.stderr parsl.dataflow.dflow DEBUG: Task 1093 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1093 submitted for App characterizeImage, waiting on task 1092 parsl.dataflow.dflow DEBUG: Task 1093 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1093 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1094 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1094 submitted for App calibrate, waiting on task 1093 parsl.dataflow.dflow DEBUG: Task 1094 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1094 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1095 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1095 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1095 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1095 has memoization hash 30b97032b67a365fc215620344d4ad93 parsl.dataflow.memoization INFO: Task 1095 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c350c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1095 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/afa20b57-8fb5-4d22-9428-380c3a479698_isr_898798_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/afa20b57-8fb5-4d22-9428-380c3a479698_isr_898798_40.stderr parsl.dataflow.dflow DEBUG: Task 1096 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1096 submitted for App characterizeImage, waiting on task 1095 parsl.dataflow.dflow DEBUG: Task 1096 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1096 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1097 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1097 submitted for App calibrate, waiting on task 1096 parsl.dataflow.dflow DEBUG: Task 1097 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1097 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1098 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1098 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1098 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1098 has memoization hash b8d2712c4cba476572f9afd1dbc60d2e parsl.dataflow.memoization INFO: Task 1098 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c350f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1098 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/f1fa8264-3fee-4c14-b46f-873c133f5f00_isr_898800_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/f1fa8264-3fee-4c14-b46f-873c133f5f00_isr_898800_50.stderr parsl.dataflow.dflow DEBUG: Task 1099 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1099 submitted for App characterizeImage, waiting on task 1098 parsl.dataflow.dflow DEBUG: Task 1099 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1099 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1100 submitted for App calibrate, waiting on task 1099 parsl.dataflow.dflow DEBUG: Task 1100 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1100 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1101 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1101 has memoization hash 185034f07822fb0a7163df0086405288 parsl.dataflow.memoization INFO: Task 1101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c351260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1101 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/61142537-2aa0-41f8-b75d-bcf562f7980e_isr_898695_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/61142537-2aa0-41f8-b75d-bcf562f7980e_isr_898695_3.stderr parsl.dataflow.dflow DEBUG: Task 1102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1102 submitted for App characterizeImage, waiting on task 1101 parsl.dataflow.dflow DEBUG: Task 1102 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1102 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1103 submitted for App calibrate, waiting on task 1102 parsl.dataflow.dflow DEBUG: Task 1103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1104 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1104 has memoization hash 06e823adbfc608f08122fe563dc54cdc parsl.dataflow.memoization INFO: Task 1104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c351580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1104 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/19d906ea-7077-40d3-94e8-d4af7b15814b_isr_898798_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/19d906ea-7077-40d3-94e8-d4af7b15814b_isr_898798_43.stderr parsl.dataflow.dflow DEBUG: Task 1105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1105 submitted for App characterizeImage, waiting on task 1104 parsl.dataflow.dflow DEBUG: Task 1105 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1105 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1106 submitted for App calibrate, waiting on task 1105 parsl.dataflow.dflow DEBUG: Task 1106 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1106 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1107 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1107 has memoization hash 904cbc93d29ed39d5bd63c2ed13141c2 parsl.dataflow.memoization INFO: Task 1107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3518a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1107 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/3c6c34a9-b96c-4030-be87-985ea9b2ee63_isr_898695_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/3c6c34a9-b96c-4030-be87-985ea9b2ee63_isr_898695_53.stderr parsl.dataflow.dflow DEBUG: Task 1108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1108 submitted for App characterizeImage, waiting on task 1107 parsl.dataflow.dflow DEBUG: Task 1108 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1108 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1109 submitted for App calibrate, waiting on task 1108 parsl.dataflow.dflow DEBUG: Task 1109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1110 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1110 has memoization hash c77733e9304c87dda0f359a9ae73509e parsl.dataflow.memoization INFO: Task 1110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c351bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1110 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/7cd34713-454c-4633-8a5d-bc671360be42_isr_898696_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/7cd34713-454c-4633-8a5d-bc671360be42_isr_898696_56.stderr parsl.dataflow.dflow DEBUG: Task 1111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1111 submitted for App characterizeImage, waiting on task 1110 parsl.dataflow.dflow DEBUG: Task 1111 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1111 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1112 submitted for App calibrate, waiting on task 1111 parsl.dataflow.dflow DEBUG: Task 1112 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1112 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1113 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1113 has memoization hash 914298d58f8f0fdd563dc4876acfc0e3 parsl.dataflow.memoization INFO: Task 1113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c351ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1113 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/9b6ebefd-22c3-4ef7-92e5-6ffd3528381c_isr_898799_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/9b6ebefd-22c3-4ef7-92e5-6ffd3528381c_isr_898799_20.stderr parsl.dataflow.dflow DEBUG: Task 1114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1114 submitted for App characterizeImage, waiting on task 1113 parsl.dataflow.dflow DEBUG: Task 1114 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1114 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1115 submitted for App calibrate, waiting on task 1114 parsl.dataflow.dflow DEBUG: Task 1115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1115 has outstanding dependencies, so launch_if_ready skipping 2024-03-13 09:45:11 proc_lsst.multi:170 [INFO] [multi] job_id 17010051 proc_lsst.multi INFO: [multi] job_id 17010051 2024-03-13 09:45:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 09:45:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:11 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010051 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010051 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17010051 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-13 09:45:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-13 09:45:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 09:45:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:11 proc_lsst.multi:170 [INFO] [multi] job_id 17010052 proc_lsst.multi INFO: [multi] job_id 17010052 2024-03-13 09:45:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 09:45:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:11 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010052 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010052 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17010052 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-13 09:45:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-13 09:45:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 09:45:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:11 proc_lsst.multi:170 [INFO] [multi] job_id 17010053 proc_lsst.multi INFO: [multi] job_id 17010053 2024-03-13 09:45:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 09:45:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:11 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010053 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010053 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17010053 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-13 09:45:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-13 09:45:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 09:45:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:11 proc_lsst.multi:170 [INFO] [multi] job_id 17010054 proc_lsst.multi INFO: [multi] job_id 17010054 2024-03-13 09:45:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 09:45:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:11 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010054 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010054 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17010054 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-13 09:45:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-13 09:45:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 09:45:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:11 proc_lsst.multi:170 [INFO] [multi] job_id 17010055 proc_lsst.multi INFO: [multi] job_id 17010055 2024-03-13 09:45:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 09:45:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:11 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010055 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010055 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17010055 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-13 09:45:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-13 09:45:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 09:45:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:12 proc_lsst.multi:170 [INFO] [multi] job_id 17010056 proc_lsst.multi INFO: [multi] job_id 17010056 2024-03-13 09:45:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 09:45:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:12 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010056 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010056 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17010056 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-13 09:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-13 09:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 09:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:12 proc_lsst.multi:170 [INFO] [multi] job_id 17010057 proc_lsst.multi INFO: [multi] job_id 17010057 2024-03-13 09:45:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 09:45:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:12 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010057 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010057 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17010057 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-13 09:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-13 09:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 09:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:13 proc_lsst.multi:170 [INFO] [multi] job_id 17010058 proc_lsst.multi INFO: [multi] job_id 17010058 2024-03-13 09:45:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 09:45:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010058 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010058 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17010058 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-13 09:45:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-13 09:45:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 09:45:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:13 proc_lsst.multi:170 [INFO] [multi] job_id 17010059 proc_lsst.multi INFO: [multi] job_id 17010059 2024-03-13 09:45:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 09:45:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010059 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010059 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17010059 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-13 09:45:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-13 09:45:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 09:45:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:14 proc_lsst.multi:170 [INFO] [multi] job_id 17010060 proc_lsst.multi INFO: [multi] job_id 17010060 2024-03-13 09:45:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 09:45:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:14 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010060 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010060 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17010060 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-13 09:45:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-13 09:45:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 09:45:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:15 proc_lsst.multi:170 [INFO] [multi] job_id 17010061 proc_lsst.multi INFO: [multi] job_id 17010061 2024-03-13 09:45:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 09:45:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010061 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010061 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17010061 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-13 09:45:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-13 09:45:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 09:45:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:16 proc_lsst.multi:170 [INFO] [multi] job_id 17010062 proc_lsst.multi INFO: [multi] job_id 17010062 2024-03-13 09:45:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 09:45:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:16 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010062 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010062 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17010062 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-13 09:45:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-13 09:45:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 09:45:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:16 proc_lsst.multi:170 [INFO] [multi] job_id 17010063 proc_lsst.multi INFO: [multi] job_id 17010063 2024-03-13 09:45:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 09:45:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:16 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010063 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010063 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17010063 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-13 09:45:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-13 09:45:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 09:45:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:17 proc_lsst.multi:170 [INFO] [multi] job_id 17010064 proc_lsst.multi INFO: [multi] job_id 17010064 2024-03-13 09:45:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 09:45:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:17 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010064 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010064 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17010064 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-13 09:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-13 09:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 09:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:17 proc_lsst.multi:170 [INFO] [multi] job_id 17010065 proc_lsst.multi INFO: [multi] job_id 17010065 2024-03-13 09:45:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:45:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:17 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010065 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010065 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17010065 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-13 09:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-13 09:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:18 proc_lsst.multi:170 [INFO] [multi] job_id 17010066 proc_lsst.multi INFO: [multi] job_id 17010066 2024-03-13 09:45:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-13 09:45:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010066 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010066 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17010066 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-13 09:45:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-13 09:45:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-13 09:45:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:19 proc_lsst.multi:170 [INFO] [multi] job_id 17010067 proc_lsst.multi INFO: [multi] job_id 17010067 2024-03-13 09:45:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-13 09:45:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010067 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010067 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17010067 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-13 09:45:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-13 09:45:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-13 09:45:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:19 proc_lsst.multi:170 [INFO] [multi] job_id 17010069 proc_lsst.multi INFO: [multi] job_id 17010069 2024-03-13 09:45:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-13 09:45:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010069 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010069 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17010069 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-13 09:45:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-13 09:45:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-13 09:45:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:20 proc_lsst.multi:170 [INFO] [multi] job_id 17010072 proc_lsst.multi INFO: [multi] job_id 17010072 2024-03-13 09:45:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-13 09:45:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010072 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010072 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17010072 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-13 09:45:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-13 09:45:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-13 09:45:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:22 proc_lsst.multi:170 [INFO] [multi] job_id 17010074 proc_lsst.multi INFO: [multi] job_id 17010074 2024-03-13 09:45:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-13 09:45:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:22 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010074 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010074 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17010074 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-13 09:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-13 09:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-13 09:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:23 proc_lsst.multi:170 [INFO] [multi] job_id 17010075 proc_lsst.multi INFO: [multi] job_id 17010075 2024-03-13 09:45:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-13 09:45:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:23 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010075 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010075 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17010075 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-13 09:45:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-13 09:45:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-13 09:45:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:24 proc_lsst.multi:170 [INFO] [multi] job_id 17010076 proc_lsst.multi INFO: [multi] job_id 17010076 2024-03-13 09:45:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-13 09:45:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:24 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010076 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010076 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17010076 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-13 09:45:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-13 09:45:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-13 09:45:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dddd78638ba754556c07704f10a6c554 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash f63645b4785dca99abc7b440a9e15059 parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d584a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 373 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/4e347ece-ab75-47ab-9837-6522855f64d3_characterizeImage_898798_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/4e347ece-ab75-47ab-9837-6522855f64d3_characterizeImage_898798_15.stderr parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/ebd4a761-135e-4742-b637-b43068a9c5b3_isr_898798_15.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/ebd4a761-135e-4742-b637-b43068a9c5b3_isr_898798_15.stderr 2024-03-13 09:45:25 proc_lsst.multi:170 [INFO] [multi] job_id 17010077 proc_lsst.multi INFO: [multi] job_id 17010077 2024-03-13 09:45:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-13 09:45:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:25 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010077 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010077 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17010077 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-13 09:45:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-13 09:45:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-13 09:45:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4721ad1d15bb1e4f21fc9b868b0624eb with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 17f33406232056a6f78ec2f524ecded5 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d584f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 374 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/7b06b4e9-8f2b-4d2e-9547-3158c6d7d943_characterizeImage_898696_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/7b06b4e9-8f2b-4d2e-9547-3158c6d7d943_characterizeImage_898696_1.stderr parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/b2e06311-10d8-4985-8c8a-3214df9703bf_isr_898696_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/b2e06311-10d8-4985-8c8a-3214df9703bf_isr_898696_1.stderr 2024-03-13 09:45:27 proc_lsst.multi:170 [INFO] [multi] job_id 17010078 proc_lsst.multi INFO: [multi] job_id 17010078 2024-03-13 09:45:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-13 09:45:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:27 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010078 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010078 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17010078 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-13 09:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-13 09:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-13 09:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:28 proc_lsst.multi:170 [INFO] [multi] job_id 17010080 proc_lsst.multi INFO: [multi] job_id 17010080 2024-03-13 09:45:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:28 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17010080 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17010080 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17010080 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-13 09:45:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-13 09:45:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 09:45:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27110a3eb26df4f7dc63a668397508ed with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash ea9caa3428e33940898d1e18f7f3783b parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d585ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 375 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/d14e9a83-2465-41cb-b854-10f0f2a645c6_characterizeImage_898798_57.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/d14e9a83-2465-41cb-b854-10f0f2a645c6_characterizeImage_898798_57.stderr parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/22ba94da-525a-4ef8-be40-7d3f703d2069_isr_898798_57.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/22ba94da-525a-4ef8-be40-7d3f703d2069_isr_898798_57.stderr 2024-03-13 09:45:29 proc_lsst.multi:170 [INFO] [multi] job_id 17010081 proc_lsst.multi INFO: [multi] job_id 17010081 2024-03-13 09:45:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:29 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17010081 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17010081 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17010081 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-13 09:45:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-13 09:45:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:31 proc_lsst.multi:170 [INFO] [multi] job_id 17010082 proc_lsst.multi INFO: [multi] job_id 17010082 2024-03-13 09:45:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 09:45:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:31 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17010082 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17010082 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17010082 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-13 09:45:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-13 09:45:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 09:45:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72b8d074b5ca9ae9dd1be881d87422ae with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 0a9b478f4c0dc98203e8d75975fd8e1b parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d585b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 376 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/fe975d71-25e4-4fd3-89ff-a290c3e63a11_characterizeImage_898694_39.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/fe975d71-25e4-4fd3-89ff-a290c3e63a11_characterizeImage_898694_39.stderr parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/03a1391e-5233-44a9-8953-c7988dcebae9_isr_898694_39.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/03a1391e-5233-44a9-8953-c7988dcebae9_isr_898694_39.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d7b8fb4370c1e0944ffce5748432410 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash a826b2acb9732fd7f36988e976647d8c parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d585760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 377 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/b687ec28-03c2-4c23-8d95-d9792c516218_characterizeImage_898798_6.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/b687ec28-03c2-4c23-8d95-d9792c516218_characterizeImage_898798_6.stderr parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/fb102970-4209-47c8-adc0-797e4a3389a8_isr_898798_6.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/fb102970-4209-47c8-adc0-797e4a3389a8_isr_898798_6.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6f9c6803c6f2e41ef764a3de27dfdf1 with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 156af6c974e7f50c0d95883c66f06b5b parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5862a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 378 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/61b94d0e-4e1b-4091-ba62-b77cecda938d_characterizeImage_898695_45.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/61b94d0e-4e1b-4091-ba62-b77cecda938d_characterizeImage_898695_45.stderr parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/0960e77e-2d9c-42d8-8ab3-81d22ea96970_isr_898695_45.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/0960e77e-2d9c-42d8-8ab3-81d22ea96970_isr_898695_45.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36802b7279daacc60c2b4ebbff5d59f9 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 3c551798e220602c838de58b53d28be7 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5872e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 379 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/bbd81449-432e-4786-bd8e-4c5934eed9e5_characterizeImage_898696_18.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/bbd81449-432e-4786-bd8e-4c5934eed9e5_characterizeImage_898696_18.stderr parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/ddd01043-da94-4d1c-8f7c-d2485b9e7f24_isr_898696_18.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/ddd01043-da94-4d1c-8f7c-d2485b9e7f24_isr_898696_18.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc30609d9a18a54a656f8ba6745a50e3 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash f76e6b3dcaeadf789603703ecdc426a6 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d586fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 380 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/ffe8e3ba-d958-4fcc-b3d0-c89100dfdbdf_characterizeImage_898799_24.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/ffe8e3ba-d958-4fcc-b3d0-c89100dfdbdf_characterizeImage_898799_24.stderr parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/a9b50873-8a87-4591-9ed1-337d960dcb8b_isr_898799_24.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/a9b50873-8a87-4591-9ed1-337d960dcb8b_isr_898799_24.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 056d08361cc28ccf8a264e446ee5a13b with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 54a5b617db774d3163e9dbd0040d6f9d parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d587600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 381 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/746d5ff3-0e87-4b04-840a-e77c768bdf3e_characterizeImage_898695_62.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/746d5ff3-0e87-4b04-840a-e77c768bdf3e_characterizeImage_898695_62.stderr parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/500359ef-514c-45ab-92ed-76c8b73d22ce_isr_898695_62.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/500359ef-514c-45ab-92ed-76c8b73d22ce_isr_898695_62.stderr 2024-03-13 09:45:32 proc_lsst.multi:170 [INFO] [multi] job_id 17010084 proc_lsst.multi INFO: [multi] job_id 17010084 2024-03-13 09:45:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 09:45:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:32 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17010084 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17010084 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17010084 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-13 09:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-13 09:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 09:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53039eb29b019d6a4f1b93b16fccb6bb with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 9b53183c7d188d64aa53806008a5ddb0 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d587920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 382 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/a49e8108-af1c-4307-8cf1-08e5e3451426_characterizeImage_898798_4.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/a49e8108-af1c-4307-8cf1-08e5e3451426_characterizeImage_898798_4.stderr parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/dc06ec55-1012-4752-b547-ad469a8f2d70_isr_898798_4.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/dc06ec55-1012-4752-b547-ad469a8f2d70_isr_898798_4.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc58fc73984aac1864486ae6e7dc56a4 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 40807a35a5c5def4c8ba29d34bc9d031 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d585440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 383 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/4783d71a-270a-4a6a-b9b8-6f8bad961d4c_characterizeImage_898696_36.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/4783d71a-270a-4a6a-b9b8-6f8bad961d4c_characterizeImage_898696_36.stderr parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/071f84b1-1478-4c33-8db2-7430d66dce7b_isr_898696_36.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/071f84b1-1478-4c33-8db2-7430d66dce7b_isr_898696_36.stderr 2024-03-13 09:45:33 proc_lsst.multi:170 [INFO] [multi] job_id 17010086 proc_lsst.multi INFO: [multi] job_id 17010086 2024-03-13 09:45:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 09:45:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:33 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17010086 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17010086 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17010086 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 35, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 1/34 running/pending blocks, and 30 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 35 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 58 more blocks parsl.executors.status_handling INFO: Scaling out by 58 blocks parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-13 09:45:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-13 09:45:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 09:45:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e075d87a00ffef96973b32525900e95a with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 478a5105c55e59d175538fb08648c6a7 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d586ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 384 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/549d0c03-35c4-47ae-a5cd-aef6687212a1_characterizeImage_898800_36.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/549d0c03-35c4-47ae-a5cd-aef6687212a1_characterizeImage_898800_36.stderr parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/461dd6ca-6d6b-42c2-8f0d-c28bb64c0a24_isr_898800_36.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/461dd6ca-6d6b-42c2-8f0d-c28bb64c0a24_isr_898800_36.stderr 2024-03-13 09:45:34 proc_lsst.multi:170 [INFO] [multi] job_id 17010087 proc_lsst.multi INFO: [multi] job_id 17010087 2024-03-13 09:45:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:34 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17010087 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17010087 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17010087 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-13 09:45:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-13 09:45:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 09:45:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:35 proc_lsst.multi:170 [INFO] [multi] job_id 17010088 proc_lsst.multi INFO: [multi] job_id 17010088 2024-03-13 09:45:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:35 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17010088 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17010088 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17010088 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-13 09:45:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-13 09:45:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:36 proc_lsst.multi:170 [INFO] [multi] job_id 17010089 proc_lsst.multi INFO: [multi] job_id 17010089 2024-03-13 09:45:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 09:45:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:36 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17010089 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17010089 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17010089 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-13 09:45:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-13 09:45:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 09:45:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:38 proc_lsst.multi:170 [INFO] [multi] job_id 17010091 proc_lsst.multi INFO: [multi] job_id 17010091 2024-03-13 09:45:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 09:45:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:38 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17010091 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17010091 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17010091 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-13 09:45:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-13 09:45:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 09:45:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b96b2264812d9f68f9d7ad61405da51 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash f45af7a8a0811341607a540f24679807 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d587c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 385 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/9df8d6f0-6829-4717-b3c1-b3ab1a72b673_characterizeImage_898694_18.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/9df8d6f0-6829-4717-b3c1-b3ab1a72b673_characterizeImage_898694_18.stderr parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/333ca9f4-c8b9-4658-a3a9-a24eccb90d9b_isr_898694_18.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/333ca9f4-c8b9-4658-a3a9-a24eccb90d9b_isr_898694_18.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80eef661c1459bfb7e7fc7b44e8bcb19 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash c7e7e5f10f04de4f25bf3e1f7040c058 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d587f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 386 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/778fc770-03a0-46f7-ad76-7cb81fdda818_characterizeImage_898799_49.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/778fc770-03a0-46f7-ad76-7cb81fdda818_characterizeImage_898799_49.stderr parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/44e8c293-3410-482e-9664-8c0fa06896f1_isr_898799_49.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/44e8c293-3410-482e-9664-8c0fa06896f1_isr_898799_49.stderr parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13782de9e644867cae7d9194119c26e6 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 51aeac3e81722f53471e17a86327dcdb parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d586980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 387 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/9449de46-25aa-4784-8a85-07e8a6950582_characterizeImage_898696_42.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/9449de46-25aa-4784-8a85-07e8a6950582_characterizeImage_898696_42.stderr parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/fefaeac6-6037-479e-a8d6-eced1f42431c_isr_898696_42.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/fefaeac6-6037-479e-a8d6-eced1f42431c_isr_898696_42.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 305fbc72445a9596c7de7ab202d74407 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash d1166b738f333ba7560807fb2971a2f6 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5865c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 388 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/0130ef3f-5e15-4576-ba86-2e69d69d6c09_characterizeImage_898798_61.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/0130ef3f-5e15-4576-ba86-2e69d69d6c09_characterizeImage_898798_61.stderr parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/6430d955-9eed-4af8-8334-a43a49db5885_isr_898798_61.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/6430d955-9eed-4af8-8334-a43a49db5885_isr_898798_61.stderr 2024-03-13 09:45:39 proc_lsst.multi:170 [INFO] [multi] job_id 17010092 proc_lsst.multi INFO: [multi] job_id 17010092 2024-03-13 09:45:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 09:45:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:39 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17010092 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17010092 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17010092 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-13 09:45:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-13 09:45:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 09:45:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12c1d79c1d617d82470a49725dd8ec88 with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash aef361fbde57b26027fd447201f6a62e parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 389 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/9f6776cb-79d7-4059-a952-a6e36d43067c_characterizeImage_898695_44.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/9f6776cb-79d7-4059-a952-a6e36d43067c_characterizeImage_898695_44.stderr parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/be1aa605-1176-49b2-be63-262138ac92b3_isr_898695_44.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/be1aa605-1176-49b2-be63-262138ac92b3_isr_898695_44.stderr 2024-03-13 09:45:40 proc_lsst.multi:170 [INFO] [multi] job_id 17010093 proc_lsst.multi INFO: [multi] job_id 17010093 2024-03-13 09:45:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:40 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17010093 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17010093 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17010093 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-13 09:45:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-13 09:45:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d08df0628c88cb430fcc6a52ad09501c with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 5f7291110b9f8d87bfb9ddb369345223 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5aae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 390 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/c6ce5c18-d2d6-4ac6-b466-5aa3149e2094_characterizeImage_898694_48.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/c6ce5c18-d2d6-4ac6-b466-5aa3149e2094_characterizeImage_898694_48.stderr parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/b34b796e-f044-4fc2-b421-7b57177e8269_isr_898694_48.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/b34b796e-f044-4fc2-b421-7b57177e8269_isr_898694_48.stderr 2024-03-13 09:45:42 proc_lsst.multi:170 [INFO] [multi] job_id 17010094 proc_lsst.multi INFO: [multi] job_id 17010094 2024-03-13 09:45:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 09:45:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:42 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17010094 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17010094 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17010094 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-13 09:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-13 09:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 09:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bff183d9f7ebe5a7fb4f138422a0633 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 9c3fce9c8311edd1083b2d151006d1f7 parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 391 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/aba1cf11-06c3-4ef4-8432-6eee53abd353_characterizeImage_898800_7.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/aba1cf11-06c3-4ef4-8432-6eee53abd353_characterizeImage_898800_7.stderr parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/ecffa768-876b-4cc4-88b8-e4c27427ca2e_isr_898800_7.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/ecffa768-876b-4cc4-88b8-e4c27427ca2e_isr_898800_7.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6659cb25abb9214fd34e2e03a099d67 with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 7228ed8c4f0093183b42112819353669 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5aa200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 392 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/6b8aad91-534e-49a5-b141-e3f3f8a5476a_characterizeImage_898800_1.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/6b8aad91-534e-49a5-b141-e3f3f8a5476a_characterizeImage_898800_1.stderr parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/f665bb70-2d5e-48ef-a435-95f7ef5510e7_isr_898800_1.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/f665bb70-2d5e-48ef-a435-95f7ef5510e7_isr_898800_1.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a8810c0ffbdc053f327f7f72e5e54bb with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 7356d73e7f6d1d85d93510ebf88b886c parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5aab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 393 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/409fdf89-7480-4668-96af-7cb33b6dd307_characterizeImage_898695_47.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/409fdf89-7480-4668-96af-7cb33b6dd307_characterizeImage_898695_47.stderr parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/6c65040a-3fa0-4ffe-9c3d-e1a20175109c_isr_898695_47.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/6c65040a-3fa0-4ffe-9c3d-e1a20175109c_isr_898695_47.stderr 2024-03-13 09:45:43 proc_lsst.multi:170 [INFO] [multi] job_id 17010095 proc_lsst.multi INFO: [multi] job_id 17010095 2024-03-13 09:45:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 09:45:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:43 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17010095 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17010095 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17010095 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-13 09:45:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-13 09:45:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 09:45:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9ceca399ac572a11d634d706af8f9c4 with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 7e53527e76b669dc9611cb7fae37ee03 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 394 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/0f78ee3a-dc59-4311-9169-1e338a576584_characterizeImage_898695_58.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/0f78ee3a-dc59-4311-9169-1e338a576584_characterizeImage_898695_58.stderr parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/0f9c9a5f-dc72-4720-be49-bac45f1e9f95_isr_898695_58.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/0f9c9a5f-dc72-4720-be49-bac45f1e9f95_isr_898695_58.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9db6bfdbf973c6932a7c077a7c9ff62f with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash a7a28519283f41eacdbfa55a69ebf6fe parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5aa840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 395 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/d86e7923-f138-475d-a237-ecd482f3996f_characterizeImage_898800_58.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/d86e7923-f138-475d-a237-ecd482f3996f_characterizeImage_898800_58.stderr parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/9ccabdfc-acff-490d-b558-37f0f851a813_isr_898800_58.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/9ccabdfc-acff-490d-b558-37f0f851a813_isr_898800_58.stderr 2024-03-13 09:45:44 proc_lsst.multi:170 [INFO] [multi] job_id 17010096 proc_lsst.multi INFO: [multi] job_id 17010096 2024-03-13 09:45:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 09:45:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:44 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17010096 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17010096 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17010096 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-13 09:45:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-13 09:45:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 09:45:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:46 proc_lsst.multi:170 [INFO] [multi] job_id 17010097 proc_lsst.multi INFO: [multi] job_id 17010097 2024-03-13 09:45:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 09:45:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:46 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17010097 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17010097 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17010097 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-13 09:45:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-13 09:45:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 09:45:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d15a98ee78ff12c4865ab252c89a5fa with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 58440635ae9edfa672b21ffcf7ddb7f5 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5abf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 396 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/39702eb8-6ac7-42f5-9775-3fb7d7d1285e_characterizeImage_898798_32.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/39702eb8-6ac7-42f5-9775-3fb7d7d1285e_characterizeImage_898798_32.stderr parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/f9c57548-e641-4ebe-99ad-1987bdcba294_isr_898798_32.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/f9c57548-e641-4ebe-99ad-1987bdcba294_isr_898798_32.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a84828b781324d98abb246ce341ed363 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 4b73b651bdda9f703d9377b25e84853f parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ab7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 397 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ddc1aba8-58a8-4a67-b61c-3002a70819cd_characterizeImage_898798_46.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ddc1aba8-58a8-4a67-b61c-3002a70819cd_characterizeImage_898798_46.stderr parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/2fa7de8b-3992-43a6-a3c5-c3345c9dc37a_isr_898798_46.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/2fa7de8b-3992-43a6-a3c5-c3345c9dc37a_isr_898798_46.stderr parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a46b2ebcc7671c9a40687fa51c5a457 with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 22718c97fb8ebc7d147ae91fc22a202e parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cc220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 398 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/b414abf4-c8f6-45cc-9c49-9a8c3c244201_characterizeImage_898694_62.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/b414abf4-c8f6-45cc-9c49-9a8c3c244201_characterizeImage_898694_62.stderr parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/40fdb079-5cbf-4151-aa47-5aafaa5d162e_isr_898694_62.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/40fdb079-5cbf-4151-aa47-5aafaa5d162e_isr_898694_62.stderr 2024-03-13 09:45:47 proc_lsst.multi:170 [INFO] [multi] job_id 17010098 proc_lsst.multi INFO: [multi] job_id 17010098 2024-03-13 09:45:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 09:45:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:47 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17010098 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17010098 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17010098 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-13 09:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-13 09:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 09:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6d1e7f0cdb2ecc73d36566e035b7481 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 4e066cbbd71f9cb2493d9f8e66fe1958 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5abb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 399 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/24f2afbb-0e97-4c00-b6c0-fc099f1a5f70_characterizeImage_898695_60.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/24f2afbb-0e97-4c00-b6c0-fc099f1a5f70_characterizeImage_898695_60.stderr parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/767ebc18-f779-4184-83b6-f6cb408f7c80_isr_898695_60.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/767ebc18-f779-4184-83b6-f6cb408f7c80_isr_898695_60.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d594d037e1d9094d158d9a2e7149bce8 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash e5342e3b1ca9f7ffc5c1a622c9cd9f5a parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5abe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 400 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/1f595f02-9b38-431f-9017-8da0297bd727_characterizeImage_898799_58.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/1f595f02-9b38-431f-9017-8da0297bd727_characterizeImage_898799_58.stderr parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/bc7fc3ee-4f74-4223-8ff7-ebdf425ed777_isr_898799_58.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/bc7fc3ee-4f74-4223-8ff7-ebdf425ed777_isr_898799_58.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1a9d92196b42d6c3e5a5610513214d6 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 0351226e6d5e7e58e1537bffb1bd4673 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ccae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 401 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/79e0dd7c-83a9-4498-83af-7520b03515a2_characterizeImage_898696_13.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/79e0dd7c-83a9-4498-83af-7520b03515a2_characterizeImage_898696_13.stderr parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/432efd01-601b-4868-bcbb-d47ceded1578_isr_898696_13.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/432efd01-601b-4868-bcbb-d47ceded1578_isr_898696_13.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56aa47348f1732e3f987d423418b2fbc with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 3a3a1e1adfb881b080886d3d130beeb7 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cc7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 402 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/852c74f9-255e-47a6-b613-ce709d7432b6_characterizeImage_898800_54.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/852c74f9-255e-47a6-b613-ce709d7432b6_characterizeImage_898800_54.stderr parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/90e7fd80-d2eb-4644-a250-95adb1e7be83_isr_898800_54.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/90e7fd80-d2eb-4644-a250-95adb1e7be83_isr_898800_54.stderr 2024-03-13 09:45:48 proc_lsst.multi:170 [INFO] [multi] job_id 17010099 proc_lsst.multi INFO: [multi] job_id 17010099 2024-03-13 09:45:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 09:45:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:48 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17010099 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17010099 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 17010099 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-13 09:45:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-13 09:45:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 09:45:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b2a43b561b8ad77c048da6cc5f4fb15 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 12a2b88c13c94b84298bd5afe6890e97 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 403 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/97c76c2d-7941-48fd-b97e-55845ce70722_characterizeImage_898695_55.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/97c76c2d-7941-48fd-b97e-55845ce70722_characterizeImage_898695_55.stderr parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/bead010d-ea01-404a-8fee-dceaa12a264f_isr_898695_55.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/bead010d-ea01-404a-8fee-dceaa12a264f_isr_898695_55.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f1b0148dd7fa71b1554d52f5f94e3d7 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 0f8cc9ed4058661154576f90c40ad5c1 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cd440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 404 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/171067a5-07a8-4c61-bd48-e499acee32f6_characterizeImage_898695_35.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/171067a5-07a8-4c61-bd48-e499acee32f6_characterizeImage_898695_35.stderr parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/dc72a31a-1e00-4b35-963a-bb1605741efe_isr_898695_35.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/dc72a31a-1e00-4b35-963a-bb1605741efe_isr_898695_35.stderr 2024-03-13 09:45:49 proc_lsst.multi:170 [INFO] [multi] job_id 17010100 proc_lsst.multi INFO: [multi] job_id 17010100 2024-03-13 09:45:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 09:45:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:49 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17010100 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17010100 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 17010100 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-13 09:45:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-13 09:45:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 09:45:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:50 proc_lsst.multi:170 [INFO] [multi] job_id 17010101 proc_lsst.multi INFO: [multi] job_id 17010101 2024-03-13 09:45:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 09:45:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:50 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17010101 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17010101 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 17010101 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-13 09:45:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-13 09:45:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 09:45:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:51 proc_lsst.multi:170 [INFO] [multi] job_id 17010102 proc_lsst.multi INFO: [multi] job_id 17010102 2024-03-13 09:45:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 09:45:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:51 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17010102 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17010102 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 17010102 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-13 09:45:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-13 09:45:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 09:45:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:45:52 proc_lsst.multi:170 [INFO] [multi] job_id 17010104 proc_lsst.multi INFO: [multi] job_id 17010104 2024-03-13 09:45:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 09:45:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:52 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17010104 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17010104 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 17010104 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-13 09:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-13 09:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 09:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19eb34e720d5871322820c9c4cc80e40 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 017d4eae42dadb22eae70b6d213a98d7 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cd760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 405 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/c48e7e81-a51f-4e96-b8cc-9e748f0f4022_characterizeImage_898694_57.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/c48e7e81-a51f-4e96-b8cc-9e748f0f4022_characterizeImage_898694_57.stderr parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/bd2d4af7-7aa3-449d-8c38-a0e0a281dbc8_isr_898694_57.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/bd2d4af7-7aa3-449d-8c38-a0e0a281dbc8_isr_898694_57.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ea853824217f1052b79b6e8cc061f35 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash d91726bd6eee250267604a65d7011cdd parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cd120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 406 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/59abbccc-be23-428b-a971-1631d98a3087_characterizeImage_898694_53.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/59abbccc-be23-428b-a971-1631d98a3087_characterizeImage_898694_53.stderr parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/e33aa75b-33e9-4463-9a9a-339c43e64d89_isr_898694_53.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/e33aa75b-33e9-4463-9a9a-339c43e64d89_isr_898694_53.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14d995634fa0c397508d8002f34e01da with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 64f7785d3720f7d0c2c5a1ad9082697d parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 407 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/a06ccafd-1c79-473b-99f4-918df2279237_characterizeImage_898694_50.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/a06ccafd-1c79-473b-99f4-918df2279237_characterizeImage_898694_50.stderr parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/e54fe647-726d-4c05-8d12-dbf029c96829_isr_898694_50.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/e54fe647-726d-4c05-8d12-dbf029c96829_isr_898694_50.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 620182c3ff1b3a3ddff7f29cb38624f5 with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 0d7933ed13c8c642d3d69e5b0d16f557 parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 408 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/93b7d5c3-a04f-4f3a-9d8d-435d24b80257_characterizeImage_898799_46.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/93b7d5c3-a04f-4f3a-9d8d-435d24b80257_characterizeImage_898799_46.stderr parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/04ad5366-d984-4c84-9a67-fd8711aa8c75_isr_898799_46.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/04ad5366-d984-4c84-9a67-fd8711aa8c75_isr_898799_46.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97b95643e162322737e037fb0fdb0dcc with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 63a950ddb84a6421892977f52690af4e parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ab1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 409 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/ba606731-dd0a-46c7-abeb-9a6b74c6ad7a_characterizeImage_898696_20.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/ba606731-dd0a-46c7-abeb-9a6b74c6ad7a_characterizeImage_898696_20.stderr parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/4a86629b-82a9-4f29-81cd-739e3717df4f_isr_898696_20.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/4a86629b-82a9-4f29-81cd-739e3717df4f_isr_898696_20.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e91fed8770239c922b96c27059342754 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 93111b41600e07a57ded45baedcd9620 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 410 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/595267a1-4cc6-4ac4-bb51-5a7a935e4ff4_characterizeImage_898694_36.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/595267a1-4cc6-4ac4-bb51-5a7a935e4ff4_characterizeImage_898694_36.stderr parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/01108099-1e3f-4d01-a3be-05b9b5237874_isr_898694_36.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/01108099-1e3f-4d01-a3be-05b9b5237874_isr_898694_36.stderr parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5949ac70238af7e8cd424530585becbf with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash f8793d37da229a72a703b1c7ada078c5 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 411 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/33acfb60-5306-4ee5-a475-63de6bd27afb_characterizeImage_898798_24.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/33acfb60-5306-4ee5-a475-63de6bd27afb_characterizeImage_898798_24.stderr parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/e010299a-81e3-43f1-975d-eb56c0a090c3_isr_898798_24.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/e010299a-81e3-43f1-975d-eb56c0a090c3_isr_898798_24.stderr parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43f2d216a6bd968d5624a88a17b5ee58 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 9ce31b63de8a1e8db5b02d67bed0cd4b parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 412 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/840b8b6c-e6a1-4ba1-94f0-7b93c9bdd100_characterizeImage_898798_53.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/840b8b6c-e6a1-4ba1-94f0-7b93c9bdd100_characterizeImage_898798_53.stderr parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/8348b90c-4d73-4e1f-a027-92f6c242af6c_isr_898798_53.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/8348b90c-4d73-4e1f-a027-92f6c242af6c_isr_898798_53.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1f8f38b32286e7fe94b4aad0e69d444 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 11566d03e09dfeff4ef92754904755c0 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 413 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/258750c0-435e-4dea-8648-d0ef781f30ca_characterizeImage_898696_41.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/258750c0-435e-4dea-8648-d0ef781f30ca_characterizeImage_898696_41.stderr parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/7eaa9d9c-2bb4-4d33-b123-7a7d3e0a86bd_isr_898696_41.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/7eaa9d9c-2bb4-4d33-b123-7a7d3e0a86bd_isr_898696_41.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10be0233bafe76c4301491f657e4b859 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash adba8059dd6abf10b89a5fe5a6faf3bf parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5aa520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 414 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/694194a0-035a-4778-93c2-7d66b7c57abc_characterizeImage_898798_52.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/694194a0-035a-4778-93c2-7d66b7c57abc_characterizeImage_898798_52.stderr parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/6c497fd7-306e-40e7-ab8b-026b8debd94d_isr_898798_52.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/6c497fd7-306e-40e7-ab8b-026b8debd94d_isr_898798_52.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4de89dbd03caf405be4c77d1237805c with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash c5c936d22a68738f38ded50be702136d parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ab4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 415 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/9ce11716-6ef1-4762-bf4e-4abd9244d1dd_characterizeImage_898799_36.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/9ce11716-6ef1-4762-bf4e-4abd9244d1dd_characterizeImage_898799_36.stderr parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/e02d2029-da94-49b6-8568-40d1bf3182ce_isr_898799_36.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/e02d2029-da94-49b6-8568-40d1bf3182ce_isr_898799_36.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3829ac15b68d38fcd5dd5ac5a4b5c206 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 5c53827cb375d858cf00705d7fd5ab5f parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 416 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/55352941-c719-46bf-a64f-86e03424aed1_characterizeImage_898694_55.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/55352941-c719-46bf-a64f-86e03424aed1_characterizeImage_898694_55.stderr parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/0fad463c-cb87-42a8-9fc6-f88af96c633c_isr_898694_55.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/0fad463c-cb87-42a8-9fc6-f88af96c633c_isr_898694_55.stderr parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 162cfd913f0cef137f2c66ead3534f64 with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash d08c1ed2fcc405922fdaea9baa6e3549 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ced40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 417 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/b534cb93-ef55-47d7-b3ce-2cd981eaad8c_characterizeImage_898800_15.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/b534cb93-ef55-47d7-b3ce-2cd981eaad8c_characterizeImage_898800_15.stderr parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/a9a4331d-d21b-419f-92dc-99761cd333e1_isr_898800_15.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/a9a4331d-d21b-419f-92dc-99761cd333e1_isr_898800_15.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f8b588649491f006e560cbdb7a37ec6 with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash cc28a2e33c42f5bc028f5215051f8427 parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ce700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 418 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/e8799074-2fd1-4814-868b-79ec584004cf_characterizeImage_898695_17.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/e8799074-2fd1-4814-868b-79ec584004cf_characterizeImage_898695_17.stderr parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/79a4f861-5b37-4c6d-9c36-9ee5e20a4d79_isr_898695_17.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/79a4f861-5b37-4c6d-9c36-9ee5e20a4d79_isr_898695_17.stderr 2024-03-13 09:45:53 proc_lsst.multi:170 [INFO] [multi] job_id 17010105 proc_lsst.multi INFO: [multi] job_id 17010105 2024-03-13 09:45:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 09:45:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:53 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17010105 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17010105 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 17010105 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-13 09:45:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-13 09:45:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 09:45:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcda30ba244e0c15b2d47fb384a788fd with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash a4bf2154eb7b8e3ce1f2ce6b5b174ac3 parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 419 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/05e04231-940f-4e68-bcf5-507976033d30_characterizeImage_898694_44.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/05e04231-940f-4e68-bcf5-507976033d30_characterizeImage_898694_44.stderr parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/33902bee-b3ba-424a-9f21-aa6654459ea5_isr_898694_44.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/33902bee-b3ba-424a-9f21-aa6654459ea5_isr_898694_44.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fe2e4361bd18139f7a6ebaaea119727 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 8b1aa58ad951726c309b90013aec2ec4 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cf060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 420 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/a30b4722-d78f-467e-ab68-6bb6c9b9c5cc_characterizeImage_898694_46.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/a30b4722-d78f-467e-ab68-6bb6c9b9c5cc_characterizeImage_898694_46.stderr parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/0e9ef9f9-1102-470b-b2f3-f698b08128a6_isr_898694_46.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/0e9ef9f9-1102-470b-b2f3-f698b08128a6_isr_898694_46.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba4293e5c6c6a56f78908079daa78531 with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 7781801a28e04d6de4dd99e0af521968 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cf9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 421 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/8c3c95d4-cc33-475d-bae3-0a55f6b85477_characterizeImage_898800_27.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/8c3c95d4-cc33-475d-bae3-0a55f6b85477_characterizeImage_898800_27.stderr parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/eb9405da-da31-4803-8355-19bff0f2a8d1_isr_898800_27.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/eb9405da-da31-4803-8355-19bff0f2a8d1_isr_898800_27.stderr 2024-03-13 09:45:54 proc_lsst.multi:170 [INFO] [multi] job_id 17010106 proc_lsst.multi INFO: [multi] job_id 17010106 2024-03-13 09:45:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 09:45:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:54 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17010106 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17010106 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 17010106 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-13 09:45:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-13 09:45:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 09:45:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd3929e86eacdcf283d5c9fd9732cd0a with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash bf4ef0c6d8dcf2aca95e172c2177720d parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cf380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 422 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/f980714d-2919-4239-bee9-e7503e49a96a_characterizeImage_898695_4.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/f980714d-2919-4239-bee9-e7503e49a96a_characterizeImage_898695_4.stderr parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/45355476-83d9-4d27-9ffd-1b869dd00fcd_isr_898695_4.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/45355476-83d9-4d27-9ffd-1b869dd00fcd_isr_898695_4.stderr 2024-03-13 09:45:56 proc_lsst.multi:170 [INFO] [multi] job_id 17010107 proc_lsst.multi INFO: [multi] job_id 17010107 2024-03-13 09:45:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 09:45:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:56 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17010107 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17010107 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 17010107 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-13 09:45:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-13 09:45:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 09:45:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 284efad1011be4f11648d9ece29f15a1 with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 2e94b93d0373c690e2ecccdfcd91ca3e parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 423 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/6c4ab54e-98ef-4232-a91d-c83df954df4b_characterizeImage_898695_25.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/6c4ab54e-98ef-4232-a91d-c83df954df4b_characterizeImage_898695_25.stderr parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/8b7363ac-e048-4509-906d-526766328889_isr_898695_25.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/8b7363ac-e048-4509-906d-526766328889_isr_898695_25.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3981bcd20ec66c6586411d6d25148e7 with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash a39b3fdae4c02622aad7758dbae4f055 parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cdda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 424 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/5f432787-f449-4c8d-82e5-8fe91f9368dd_characterizeImage_898694_47.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/5f432787-f449-4c8d-82e5-8fe91f9368dd_characterizeImage_898694_47.stderr parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/f0f96198-001a-4edf-bdf2-acd5b50859c1_isr_898694_47.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/f0f96198-001a-4edf-bdf2-acd5b50859c1_isr_898694_47.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02db1b7797f84d3f3d35239769946ee0 with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash ffd643eefe09c50fb5a8d367d5d8a713 parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cfe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 425 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/bb5eb107-4898-4b6d-bb02-83aa669e3d94_characterizeImage_898800_31.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/bb5eb107-4898-4b6d-bb02-83aa669e3d94_characterizeImage_898800_31.stderr parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/c7b5407e-9c09-43f3-ba55-0d8acb6fd95b_isr_898800_31.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/c7b5407e-9c09-43f3-ba55-0d8acb6fd95b_isr_898800_31.stderr 2024-03-13 09:45:57 proc_lsst.multi:170 [INFO] [multi] job_id 17010109 proc_lsst.multi INFO: [multi] job_id 17010109 2024-03-13 09:45:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 09:45:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:57 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17010109 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17010109 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 17010109 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-13 09:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-13 09:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 09:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4788002c0e167b07e9d9c83c660336d2 with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 3885b107a5995ab1810bef4bdb83e233 parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ec680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 426 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/7d034d54-acbf-48a5-bfd8-81626a5d0dd5_characterizeImage_898798_33.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/7d034d54-acbf-48a5-bfd8-81626a5d0dd5_characterizeImage_898798_33.stderr parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/217639e6-bc7f-46e2-9ebf-57221adbd217_isr_898798_33.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/217639e6-bc7f-46e2-9ebf-57221adbd217_isr_898798_33.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59ded46f6d7d87880d82152a958f7f7a with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 66bf19c617b9f95b214636c4cdf757b9 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cf6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 427 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/2ec118bb-c4df-4ff4-b396-0b8c250eb660_characterizeImage_898695_51.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/2ec118bb-c4df-4ff4-b396-0b8c250eb660_characterizeImage_898695_51.stderr parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/f07d52b1-141d-4f71-afc2-206e2125a55d_isr_898695_51.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/f07d52b1-141d-4f71-afc2-206e2125a55d_isr_898695_51.stderr 2024-03-13 09:45:58 proc_lsst.multi:170 [INFO] [multi] job_id 17010110 proc_lsst.multi INFO: [multi] job_id 17010110 2024-03-13 09:45:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:45:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:58 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17010110 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17010110 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 17010110 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-13 09:45:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-13 09:45:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:45:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:58 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 09:45:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:45:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0284393415845ee508d706696c49eeb2 with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 5346155ea9c7580051a40bb1ef4eaaac parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ec9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 428 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/73c4918a-187f-4be1-9611-bd1abfc5ccb8_characterizeImage_898800_24.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/73c4918a-187f-4be1-9611-bd1abfc5ccb8_characterizeImage_898800_24.stderr parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/2ac7b472-aa0d-47cc-b73b-d8987e10bad4_isr_898800_24.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/2ac7b472-aa0d-47cc-b73b-d8987e10bad4_isr_898800_24.stderr parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d219ec03d9fb66d60af1a3f59029cd82 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 05e2d1f0cd130a1c42243e02d20de459 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cfce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 429 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/6ca1e530-13fa-4c33-833c-6568428a18ce_characterizeImage_898696_52.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/6ca1e530-13fa-4c33-833c-6568428a18ce_characterizeImage_898696_52.stderr parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/acd147a1-6f80-4dd2-81eb-b6c6b1bbda6a_isr_898696_52.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/acd147a1-6f80-4dd2-81eb-b6c6b1bbda6a_isr_898696_52.stderr 2024-03-13 09:45:59 proc_lsst.multi:170 [INFO] [multi] job_id 17010111 proc_lsst.multi INFO: [multi] job_id 17010111 2024-03-13 09:45:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:45:59 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010111 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010111 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 17010111 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-13 09:45:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-13 09:45:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:45:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:45:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:45:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:45:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:45:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:45:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:45:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:45:59 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:45:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:45:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:45:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ee8e5116410f9555d6286ed9a3f8410 with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash e4d6ce57811994fe9a456bb11653fdbd parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5edc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 430 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/ae0e9aa1-5eab-4184-a6fe-35869c139b00_characterizeImage_898694_31.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/ae0e9aa1-5eab-4184-a6fe-35869c139b00_characterizeImage_898694_31.stderr parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/93d98c56-b6b0-46d1-818e-8a890f91f2a6_isr_898694_31.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/93d98c56-b6b0-46d1-818e-8a890f91f2a6_isr_898694_31.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb37d29d10305314a55de599e8139c92 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash a5cd93403189d5ee5111193205696724 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ec0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 431 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/ff851375-ed26-428a-a051-c1d67a1b01fe_characterizeImage_898694_29.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/ff851375-ed26-428a-a051-c1d67a1b01fe_characterizeImage_898694_29.stderr parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/2e9c7c9c-80df-4eba-af8c-31df9385fb4b_isr_898694_29.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/2e9c7c9c-80df-4eba-af8c-31df9385fb4b_isr_898694_29.stderr 2024-03-13 09:46:00 proc_lsst.multi:170 [INFO] [multi] job_id 17010112 proc_lsst.multi INFO: [multi] job_id 17010112 2024-03-13 09:46:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 09:46:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010112 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010112 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 17010112 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-13 09:46:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-13 09:46:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 09:46:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b783d7f2b854d35f93c76c773f250625 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 82cfc3c5210f881400db597d5a17e203 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ce0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 432 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/4f269a1b-e601-47d7-b2fe-1926d46d4d37_characterizeImage_898800_40.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/4f269a1b-e601-47d7-b2fe-1926d46d4d37_characterizeImage_898800_40.stderr parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/50415f15-e2c6-4ee7-9948-797c2d2729ae_isr_898800_40.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/50415f15-e2c6-4ee7-9948-797c2d2729ae_isr_898800_40.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f24e0266c101ab5683cf54e2e2d9697b with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 902f28757ee042fd668c467bd2819eb1 parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ce3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 433 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/ad5c3957-f1d5-49e9-aed7-fbd35ce9bebf_characterizeImage_898799_10.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/ad5c3957-f1d5-49e9-aed7-fbd35ce9bebf_characterizeImage_898799_10.stderr parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/0f656dc7-4c66-4a89-91a4-4bc06e21ecb1_isr_898799_10.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/0f656dc7-4c66-4a89-91a4-4bc06e21ecb1_isr_898799_10.stderr parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fcd3bb5bca48e2415fe3324f86fb0fc with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash e497cba56f4d44e443f6491ae4f16f25 parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ee2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 434 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/57e3faad-3c33-4bf4-9483-c00249722cb9_characterizeImage_898694_5.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/57e3faad-3c33-4bf4-9483-c00249722cb9_characterizeImage_898694_5.stderr parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/38a5576c-b5cc-4dc0-a983-b416b6697b46_isr_898694_5.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/38a5576c-b5cc-4dc0-a983-b416b6697b46_isr_898694_5.stderr parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27b4c13eeafc24f1d07b73b579ce6795 with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash f7235efac539c7de40a3750651ce9588 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5edf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 435 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/b30939dc-db14-44f9-838e-a275d60ac4c0_characterizeImage_898695_18.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/b30939dc-db14-44f9-838e-a275d60ac4c0_characterizeImage_898695_18.stderr parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/415d2eed-39c3-4c69-afcb-d948a123e5fb_isr_898695_18.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/415d2eed-39c3-4c69-afcb-d948a123e5fb_isr_898695_18.stderr parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 567b582a4ce2911b310a322fc0a5a558 with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 90064832ab7598e07791d37e0e268956 parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5eec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 436 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/c2ddae44-93f6-4afd-a19f-24f55955f00e_characterizeImage_898798_13.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/c2ddae44-93f6-4afd-a19f-24f55955f00e_characterizeImage_898798_13.stderr parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/d501d583-8718-4e20-8f9a-be5d228f6823_isr_898798_13.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/d501d583-8718-4e20-8f9a-be5d228f6823_isr_898798_13.stderr 2024-03-13 09:46:02 proc_lsst.multi:170 [INFO] [multi] job_id 17010113 proc_lsst.multi INFO: [multi] job_id 17010113 2024-03-13 09:46:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 09:46:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:02 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010113 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010113 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 17010113 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-13 09:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-13 09:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:02 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 09:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:46:03 proc_lsst.multi:170 [INFO] [multi] job_id 17010114 proc_lsst.multi INFO: [multi] job_id 17010114 2024-03-13 09:46:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 09:46:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010114 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010114 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 17010114 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-13 09:46:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-13 09:46:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 09:46:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e89498a2ab2404314fa6bf33240e3536 with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 6b78d2e12ebfb25511c61ad4f76b9a5f parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ef240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 437 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/082c8ade-5624-4fbb-8091-aeac9edc6616_characterizeImage_898799_23.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/082c8ade-5624-4fbb-8091-aeac9edc6616_characterizeImage_898799_23.stderr parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/21ddfde0-e67e-4bed-9a0e-11576762d1eb_isr_898799_23.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/21ddfde0-e67e-4bed-9a0e-11576762d1eb_isr_898799_23.stderr parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e6fdd9211acd29caa18a0486aac280e with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash bd169fcb73b46fb70e0cf782149ff958 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5eef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 438 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/dd3d308e-4d92-428f-b494-4693af80b880_characterizeImage_898694_60.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/dd3d308e-4d92-428f-b494-4693af80b880_characterizeImage_898694_60.stderr parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/5046e09e-0387-4a52-9fcb-915aefdceeb0_isr_898694_60.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/5046e09e-0387-4a52-9fcb-915aefdceeb0_isr_898694_60.stderr parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a9a629d36b8ffd67e5b4e7e38f6b048 with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 8195b1a3fee85fc51d30e88d88ced5e2 parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5efec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 439 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/2567e18b-c380-43b5-85d4-b32777321532_characterizeImage_898798_41.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/2567e18b-c380-43b5-85d4-b32777321532_characterizeImage_898798_41.stderr parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/fbfffb24-4a12-4f0a-a34a-c7d916f930e5_isr_898798_41.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/fbfffb24-4a12-4f0a-a34a-c7d916f930e5_isr_898798_41.stderr parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a589b9eaa0b06841f7efa931efc78db with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 58828e96268da24db82213bc3794815a parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ef560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 440 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/88df0517-1992-4243-aebe-74b9cc2d561a_characterizeImage_898799_4.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/88df0517-1992-4243-aebe-74b9cc2d561a_characterizeImage_898799_4.stderr parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/aae9d5a4-7741-4c09-ac49-7ed2200020ed_isr_898799_4.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/aae9d5a4-7741-4c09-ac49-7ed2200020ed_isr_898799_4.stderr parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62ba6aac74a7b005e340bddbffbec664 with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 02989870e1b241773fa38c36b1b9756a parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5efba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 441 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/3c13976b-d5a2-44b3-9c24-8c2a848a4d9d_characterizeImage_898800_12.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/3c13976b-d5a2-44b3-9c24-8c2a848a4d9d_characterizeImage_898800_12.stderr parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/ba3cc9cb-6e90-414f-b91b-39a50a18beca_isr_898800_12.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/ba3cc9cb-6e90-414f-b91b-39a50a18beca_isr_898800_12.stderr 2024-03-13 09:46:04 proc_lsst.multi:170 [INFO] [multi] job_id 17010115 proc_lsst.multi INFO: [multi] job_id 17010115 2024-03-13 09:46:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010115 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010115 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 17010115 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-13 09:46:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-13 09:46:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:46:05 proc_lsst.multi:170 [INFO] [multi] job_id 17010116 proc_lsst.multi INFO: [multi] job_id 17010116 2024-03-13 09:46:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 09:46:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010116 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010116 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 17010116 parsl.executors.status_handling INFO: Allocated block ID 62 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62 2024-03-13 09:46:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 2024-03-13 09:46:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 09:46:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7247ecdac63873b3eaf1e1eeee5e0e15 with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash aaefa86c8314ae4c1be5230d360abe77 parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d608ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 442 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/a91461ad-3730-4ccd-81a7-8a82038a38a0_characterizeImage_898799_2.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/a91461ad-3730-4ccd-81a7-8a82038a38a0_characterizeImage_898799_2.stderr parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/8f846a96-bfdc-41d7-a64a-1c7516546aba_isr_898799_2.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/8f846a96-bfdc-41d7-a64a-1c7516546aba_isr_898799_2.stderr 2024-03-13 09:46:06 proc_lsst.multi:170 [INFO] [multi] job_id 17010117 proc_lsst.multi INFO: [multi] job_id 17010117 2024-03-13 09:46:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 09:46:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010117 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010117 parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 17010117 parsl.executors.status_handling INFO: Allocated block ID 63 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63 2024-03-13 09:46:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 2024-03-13 09:46:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 09:46:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ee4de340a41398f9435513edca1026e with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash e0318b0119f1f8bbe0839f7f4e78ead2 parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d609800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 443 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/6abe174d-4883-4a2f-832e-1ba138e39c3e_characterizeImage_898799_32.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/6abe174d-4883-4a2f-832e-1ba138e39c3e_characterizeImage_898799_32.stderr parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/cbe58183-d846-43c0-a966-3d898f74fb6c_isr_898799_32.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/cbe58183-d846-43c0-a966-3d898f74fb6c_isr_898799_32.stderr parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 589eb1fcce48cc93de03f51ff0cdf9d5 with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash fd8f97d421afc5c28d1d8c0b519784d3 parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6091c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 444 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/bc488d99-7876-42b4-bc90-e5b8c6282f5a_characterizeImage_898694_41.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/bc488d99-7876-42b4-bc90-e5b8c6282f5a_characterizeImage_898694_41.stderr parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/bcbf8fd9-60cb-42df-ae7a-e79a02485a76_isr_898694_41.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/bcbf8fd9-60cb-42df-ae7a-e79a02485a76_isr_898694_41.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 504de62121f961743d4b452a2960cd58 with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash f8ca691e9ad71f1feee3946b6fd92372 parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d609b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 445 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/2f9bcff7-43eb-47ad-a327-cbc316d4b671_characterizeImage_898798_23.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/2f9bcff7-43eb-47ad-a327-cbc316d4b671_characterizeImage_898798_23.stderr parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/3cbcbf24-c16c-4e57-a67e-3485d4e316ea_isr_898798_23.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/3cbcbf24-c16c-4e57-a67e-3485d4e316ea_isr_898798_23.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fb33769887e703097537b3761493a7f with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 5d42d0891d60c5f3b2847504398757b2 parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 446 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/1429c4df-06f9-4ff7-9a76-fda914202966_characterizeImage_898694_3.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/1429c4df-06f9-4ff7-9a76-fda914202966_characterizeImage_898694_3.stderr parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/d9e45d5c-05bb-40f5-a869-37e8a52141f6_isr_898694_3.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/d9e45d5c-05bb-40f5-a869-37e8a52141f6_isr_898694_3.stderr 2024-03-13 09:46:08 proc_lsst.multi:170 [INFO] [multi] job_id 17010118 proc_lsst.multi INFO: [multi] job_id 17010118 2024-03-13 09:46:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 09:46:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010118 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010118 parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 17010118 parsl.executors.status_handling INFO: Allocated block ID 64 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64 2024-03-13 09:46:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 2024-03-13 09:46:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 09:46:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae9e4a25770551756166d55c1a6952b3 with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash 05e5d819800eb778c47782406b81ecfd parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 447 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/5b5c2708-e781-4ba7-b5af-d25ac262c191_characterizeImage_898800_46.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/5b5c2708-e781-4ba7-b5af-d25ac262c191_characterizeImage_898800_46.stderr parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/36d7c93d-b7a4-4d33-95cf-ac90330d481f_isr_898800_46.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/36d7c93d-b7a4-4d33-95cf-ac90330d481f_isr_898800_46.stderr parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f86f5799f143cd68475e4a8cfe6bdf2 with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash 1491530434f7c97ad515d90a8c350782 parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 448 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/9379e340-8cfa-41b7-a84c-9d4eeaa119d6_characterizeImage_898799_18.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/9379e340-8cfa-41b7-a84c-9d4eeaa119d6_characterizeImage_898799_18.stderr parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/9fb7bb20-5c8c-4d19-b003-9b339baf98df_isr_898799_18.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/9fb7bb20-5c8c-4d19-b003-9b339baf98df_isr_898799_18.stderr parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry beafc005887a49b15d216862ee6e2dd1 with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash b8690543670a892931ddc48e9e73dae1 parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d608b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 449 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/21f27cea-9879-4c0f-93d6-d1586feca95f_characterizeImage_898800_18.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/21f27cea-9879-4c0f-93d6-d1586feca95f_characterizeImage_898800_18.stderr parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/70184483-6012-478f-b865-a7d4f72924fa_isr_898800_18.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/70184483-6012-478f-b865-a7d4f72924fa_isr_898800_18.stderr parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a7881f538950abfe58032f394f67f7d with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash b4daac0cac00a2083b368424bc78472d parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6094e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 450 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/9bdaa6db-cbb2-4d21-98b0-d5e25045cdfb_characterizeImage_898694_42.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/9bdaa6db-cbb2-4d21-98b0-d5e25045cdfb_characterizeImage_898694_42.stderr parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/099eaac4-c208-4fe8-aafd-92229ac3b25d_isr_898694_42.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/099eaac4-c208-4fe8-aafd-92229ac3b25d_isr_898694_42.stderr parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb8864a6aad2bf68cad2dc460f029d82 with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash 5fe9a56f9fa23c1feb9a606c689356b8 parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 451 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/d1956f79-6612-47e6-8b35-a11a603f780b_characterizeImage_898800_57.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/d1956f79-6612-47e6-8b35-a11a603f780b_characterizeImage_898800_57.stderr parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/1c599cda-09f8-4f73-9ac0-d72b2d001e36_isr_898800_57.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/1c599cda-09f8-4f73-9ac0-d72b2d001e36_isr_898800_57.stderr parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a6e709d82ae0ec265bd7512fc2ae308 with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash b83682532f2d74783c4b9c6f49e8bc33 parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 452 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/706e310d-5e2a-4a1b-abc3-c6f874be370c_characterizeImage_898694_10.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/706e310d-5e2a-4a1b-abc3-c6f874be370c_characterizeImage_898694_10.stderr parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/31993d28-3800-4e3f-9fb6-63f3d4d617cc_isr_898694_10.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/31993d28-3800-4e3f-9fb6-63f3d4d617cc_isr_898694_10.stderr parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba09f6f78575739e856c05aae3f20a6f with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 77df07d484b7fea77901ec1f476f1fbc parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 453 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/c2929b31-c0de-491c-88ff-9ea32bdce3ed_characterizeImage_898696_35.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/c2929b31-c0de-491c-88ff-9ea32bdce3ed_characterizeImage_898696_35.stderr parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/1892b3a4-7ab4-4bff-8c71-d2b0a2d159c0_isr_898696_35.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/1892b3a4-7ab4-4bff-8c71-d2b0a2d159c0_isr_898696_35.stderr 2024-03-13 09:46:09 proc_lsst.multi:170 [INFO] [multi] job_id 17010119 proc_lsst.multi INFO: [multi] job_id 17010119 2024-03-13 09:46:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 09:46:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010119 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010119 parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 17010119 parsl.executors.status_handling INFO: Allocated block ID 65 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65 2024-03-13 09:46:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 2024-03-13 09:46:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 09:46:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff84ceae97ac18903ec025452f3494eb with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash b7092cc857ba5385dc97a22ddc8b5d1b parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 454 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/59a24470-1455-4a8c-95ac-b228e51dc40b_characterizeImage_898799_52.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/59a24470-1455-4a8c-95ac-b228e51dc40b_characterizeImage_898799_52.stderr parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/cf5d0c2b-a256-4950-9626-9e98bee6cf14_isr_898799_52.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/cf5d0c2b-a256-4950-9626-9e98bee6cf14_isr_898799_52.stderr parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d7d72916f44a3dced4f34665e671e2e with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 6657f0ac257a6785d6838440576f4987 parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d609e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 455 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/49ea2301-66ce-4f1f-9905-e2346711a880_characterizeImage_898695_29.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/49ea2301-66ce-4f1f-9905-e2346711a880_characterizeImage_898695_29.stderr parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/bcf59e8b-04ca-4e6f-a2ca-0f66228b9e0b_isr_898695_29.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/bcf59e8b-04ca-4e6f-a2ca-0f66228b9e0b_isr_898695_29.stderr parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1ae022506a5912de8a2748c150d97c6 with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash 4b3ee609a840e6de67f57c4027c793d2 parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 456 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/36fbb940-b7a5-4d69-ac6f-5f9c08c5ef00_characterizeImage_898799_47.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/36fbb940-b7a5-4d69-ac6f-5f9c08c5ef00_characterizeImage_898799_47.stderr parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/bd93033b-2e96-4ab4-9215-4885aeb9bc04_isr_898799_47.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/bd93033b-2e96-4ab4-9215-4885aeb9bc04_isr_898799_47.stderr parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fdf5bde54fb6ee92b8c1dc0b634f908 with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash 14d4c28bb5ebde2568fff6e781bfaa3f parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 457 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/16aad049-f9e6-47c1-b997-b61c2f7884a1_characterizeImage_898798_54.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/16aad049-f9e6-47c1-b997-b61c2f7884a1_characterizeImage_898798_54.stderr parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/ac348d27-485c-4315-a55c-12ddfbd688e6_isr_898798_54.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/ac348d27-485c-4315-a55c-12ddfbd688e6_isr_898798_54.stderr parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb5208ea1b6642b3cec92d48e0a60743 with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 3b60167d21da521478ed68353a61cb5b parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 458 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/6087a1a6-7469-4768-a966-02c72ac53852_characterizeImage_898800_60.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/6087a1a6-7469-4768-a966-02c72ac53852_characterizeImage_898800_60.stderr parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/8b79a977-4446-459a-be22-54db5a785cd8_isr_898800_60.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/8b79a977-4446-459a-be22-54db5a785cd8_isr_898800_60.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d3069fa279c4cc447967e78fbbdde8d with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 0f5dbd0065a9fe5fbdb455195b5def88 parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 459 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/caadaaf8-6f9e-4b48-b49b-2337836a1003_characterizeImage_898799_22.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/caadaaf8-6f9e-4b48-b49b-2337836a1003_characterizeImage_898799_22.stderr parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/a5d0bbfb-87ac-4b11-9699-1bdcb6e79933_isr_898799_22.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/a5d0bbfb-87ac-4b11-9699-1bdcb6e79933_isr_898799_22.stderr 2024-03-13 09:46:10 proc_lsst.multi:170 [INFO] [multi] job_id 17010120 proc_lsst.multi INFO: [multi] job_id 17010120 2024-03-13 09:46:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 09:46:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010120 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010120 parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 17010120 parsl.executors.status_handling INFO: Allocated block ID 66 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66 2024-03-13 09:46:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 2024-03-13 09:46:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 09:46:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd2779e5f74cf2c980aaa93cfd3e3093 with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 810ff313ba075f5452e9ee35aa7cba07 parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 460 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/ad417399-876c-4066-8c82-f14945c7fd0a_characterizeImage_898694_54.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/ad417399-876c-4066-8c82-f14945c7fd0a_characterizeImage_898694_54.stderr parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/52cbf908-57ce-49d2-bbdf-01664b9e7b3e_isr_898694_54.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/52cbf908-57ce-49d2-bbdf-01664b9e7b3e_isr_898694_54.stderr parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a921de290bf37e5261dbed1e1ce9e1b8 with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash 50e9032ebad964bc9f2bbaa6b607549f parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 461 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/6154a904-5398-4d5e-b3b0-b68ca859b50d_characterizeImage_898694_58.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/6154a904-5398-4d5e-b3b0-b68ca859b50d_characterizeImage_898694_58.stderr parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/813ed576-93b8-4b5f-8190-d265fc2dc6c0_isr_898694_58.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/813ed576-93b8-4b5f-8190-d265fc2dc6c0_isr_898694_58.stderr 2024-03-13 09:46:11 proc_lsst.multi:170 [INFO] [multi] job_id 17010122 proc_lsst.multi INFO: [multi] job_id 17010122 2024-03-13 09:46:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 09:46:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:11 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010122 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010122 parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 17010122 parsl.executors.status_handling INFO: Allocated block ID 67 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67 2024-03-13 09:46:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 2024-03-13 09:46:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:11 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 09:46:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb8846e01eaeac367644b3c3e0f42ed5 with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash 424136afcaa41ddfea55e9669cf9c63d parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 462 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/ac40a5bd-c079-4198-acde-70e297b5af3a_characterizeImage_898696_26.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/ac40a5bd-c079-4198-acde-70e297b5af3a_characterizeImage_898696_26.stderr parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/d4763365-540c-4c1d-b593-d2112f23a8d6_isr_898696_26.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/d4763365-540c-4c1d-b593-d2112f23a8d6_isr_898696_26.stderr 2024-03-13 09:46:12 proc_lsst.multi:170 [INFO] [multi] job_id 17010123 proc_lsst.multi INFO: [multi] job_id 17010123 2024-03-13 09:46:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 09:46:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010123 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010123 parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 17010123 parsl.executors.status_handling INFO: Allocated block ID 68 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68 2024-03-13 09:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 2024-03-13 09:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 09:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebdba6361011041646ad1514594ed46b with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash b0efeed125ed8a35b317a446da53f525 parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 463 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/ed61939e-7bf4-4605-8285-b41df14ef57c_characterizeImage_898694_14.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/ed61939e-7bf4-4605-8285-b41df14ef57c_characterizeImage_898694_14.stderr parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/2a58b74a-c610-4b10-9ed9-88aba1d6115d_isr_898694_14.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/2a58b74a-c610-4b10-9ed9-88aba1d6115d_isr_898694_14.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf0a424849e63e43acd02facbb06d9f3 with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 16070d339a581124a5b4a6588ac9dc5f parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5efc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 464 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/65b2b50e-6819-4b89-8fea-413b4184befe_characterizeImage_898696_21.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/65b2b50e-6819-4b89-8fea-413b4184befe_characterizeImage_898696_21.stderr parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/c997ef57-3a46-4752-8a97-58a4cf7e4ec4_isr_898696_21.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/c997ef57-3a46-4752-8a97-58a4cf7e4ec4_isr_898696_21.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c3f1ec22574722b6bb3e0110e4ca34c with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 5dd3e2eefffd04cc870b7f4796578526 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ee8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 465 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/d35048fd-afbe-4dae-b758-499d05ab9fbd_characterizeImage_898799_8.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/d35048fd-afbe-4dae-b758-499d05ab9fbd_characterizeImage_898799_8.stderr parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/ff3c5cc2-984a-4975-8d75-fd05f0eec56f_isr_898799_8.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/ff3c5cc2-984a-4975-8d75-fd05f0eec56f_isr_898799_8.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c083dec7f3cf5147faa0c2daf945fb0e with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash d847158c2f7d8470b2fb6470e9e3f5a2 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5eccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 466 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/a8ad69e8-5f8e-4c21-87f9-27f0ec9238a0_characterizeImage_898800_41.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/a8ad69e8-5f8e-4c21-87f9-27f0ec9238a0_characterizeImage_898800_41.stderr parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/802287e0-c245-4765-9cb6-abf69986e5fe_isr_898800_41.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/802287e0-c245-4765-9cb6-abf69986e5fe_isr_898800_41.stderr parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4c95bd1c45381ae26c712feacb099ba with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash fe2947ffa00d6e1fbde3a80299677b8c parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ed940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 467 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/88f83bee-26c0-4188-9562-ef260a2ca606_characterizeImage_898799_37.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/88f83bee-26c0-4188-9562-ef260a2ca606_characterizeImage_898799_37.stderr parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/9ce3da53-bfcf-458b-a0da-1bab2002ccd2_isr_898799_37.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/9ce3da53-bfcf-458b-a0da-1bab2002ccd2_isr_898799_37.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23a8e417daa4de8d4705a465581ff178 with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash debd4bccdafafa4fe6f893187e6d1a5e parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ed300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 468 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/a5ce4e97-5c9f-49b9-8b69-b48bfe1e953f_characterizeImage_898696_5.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/a5ce4e97-5c9f-49b9-8b69-b48bfe1e953f_characterizeImage_898696_5.stderr parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/6c069d3f-4059-441f-ba8d-f2e6d654e038_isr_898696_5.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/6c069d3f-4059-441f-ba8d-f2e6d654e038_isr_898696_5.stderr parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fabff2bb6db9c5d1cc2d4618e33fbf4 with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 868d3212c27a263337649cf308b17574 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ecfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 469 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/3c4cd1f3-c8be-498e-a6ce-b81301f73d2c_characterizeImage_898695_48.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/3c4cd1f3-c8be-498e-a6ce-b81301f73d2c_characterizeImage_898695_48.stderr parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/2bf0ec7c-5e83-4cbe-a37e-ee51caf3034d_isr_898695_48.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/2bf0ec7c-5e83-4cbe-a37e-ee51caf3034d_isr_898695_48.stderr parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3e3846130926c80f5c12b69ce5f61a4 with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 307e0af186aa73805d3f884933b67d6e parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d608540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 470 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/2d9daff9-a0d6-413e-928c-96561de15d71_characterizeImage_898799_42.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/2d9daff9-a0d6-413e-928c-96561de15d71_characterizeImage_898799_42.stderr parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/94ca82bd-1921-4d02-a2cc-e610008c0956_isr_898799_42.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/94ca82bd-1921-4d02-a2cc-e610008c0956_isr_898799_42.stderr parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cf18bf6d38b96e649ecfa8a4ada1929 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 5dd10254b69b3f7caaffa174e0d6bd0d parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ee5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 471 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/0ba9eeb6-f72e-4fb8-8221-87b399700808_characterizeImage_898695_41.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/0ba9eeb6-f72e-4fb8-8221-87b399700808_characterizeImage_898695_41.stderr parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/cbc3ebad-eada-4d1a-92d3-99de91fda053_isr_898695_41.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/cbc3ebad-eada-4d1a-92d3-99de91fda053_isr_898695_41.stderr parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46626ecfad57315e6464e263d48cc79a with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 7e98a86d25497cd0f2925447d17e749e parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d608860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 472 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/738671f4-71d6-440e-b76d-d0b61b151e85_characterizeImage_898695_39.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/738671f4-71d6-440e-b76d-d0b61b151e85_characterizeImage_898695_39.stderr parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/747b8428-1613-45b7-8eff-d5dc5f5fabba_isr_898695_39.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/747b8428-1613-45b7-8eff-d5dc5f5fabba_isr_898695_39.stderr parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43628b794b317243b590790212133035 with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash a1fb2b13946de12363647c40ba5f21fc parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ef880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 473 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/b20aa47c-1b01-4f0d-b4dd-efdddaf10838_characterizeImage_898798_10.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/b20aa47c-1b01-4f0d-b4dd-efdddaf10838_characterizeImage_898798_10.stderr parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/f0ceac77-3fab-437a-9e27-0831f6bf4d62_isr_898798_10.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/f0ceac77-3fab-437a-9e27-0831f6bf4d62_isr_898798_10.stderr parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0fb715dc109fd626bb16399e56441e0 with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash a047d710faccf44546a0632ec5ebb88e parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ed620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 474 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/6f7c3096-33bf-436b-98ea-851b72f648ba_characterizeImage_898696_55.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/6f7c3096-33bf-436b-98ea-851b72f648ba_characterizeImage_898696_55.stderr parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/5f3a6e0e-61c7-4413-857e-0603b888709e_isr_898696_55.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/5f3a6e0e-61c7-4413-857e-0603b888709e_isr_898696_55.stderr parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e88e42d46fe0dc34c400d42260c3070a with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 77d7575d667b9ad34c090c8f1676287a parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 475 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/645d169e-9684-4549-8e89-89431d6f7a50_characterizeImage_898696_2.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/645d169e-9684-4549-8e89-89431d6f7a50_characterizeImage_898696_2.stderr parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/72f31e36-6f13-477d-b3f5-b388a83f4e3e_isr_898696_2.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/72f31e36-6f13-477d-b3f5-b388a83f4e3e_isr_898696_2.stderr parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfe3c0296f98489ec2f77171adbc2ade with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash f13230a5928c6f332491c38437a33b53 parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 476 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/9196f4db-d2e5-4cb1-81ec-8a08791adda2_characterizeImage_898696_59.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/9196f4db-d2e5-4cb1-81ec-8a08791adda2_characterizeImage_898696_59.stderr parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/f66c1442-4eb1-4ff7-8b3f-e1ad483199f3_isr_898696_59.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/f66c1442-4eb1-4ff7-8b3f-e1ad483199f3_isr_898696_59.stderr parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 479f3a419104edc74176f6c58a829e0c with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 54cf608ffd961b625fcbf7dd8800c3f0 parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 477 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/330a9d21-ee30-42c6-8c23-16ebc5ce9528_characterizeImage_898799_25.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/330a9d21-ee30-42c6-8c23-16ebc5ce9528_characterizeImage_898799_25.stderr parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/1f11f0b3-4243-45f9-a7f2-50a948edba27_isr_898799_25.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/1f11f0b3-4243-45f9-a7f2-50a948edba27_isr_898799_25.stderr 2024-03-13 09:46:14 proc_lsst.multi:170 [INFO] [multi] job_id 17010124 proc_lsst.multi INFO: [multi] job_id 17010124 2024-03-13 09:46:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 09:46:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:14 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010124 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010124 parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 17010124 parsl.executors.status_handling INFO: Allocated block ID 69 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69 2024-03-13 09:46:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 2024-03-13 09:46:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:14 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 09:46:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e048e3218ad058a92a9e66d35302304c with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash d09b21d6ab07e0164bc62f6ffbf3a9a6 parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 478 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/592c88e9-5f66-4730-9495-af7ba3b0bc9e_characterizeImage_898799_45.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/592c88e9-5f66-4730-9495-af7ba3b0bc9e_characterizeImage_898799_45.stderr parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/9165cfcd-7e59-44bb-8cb5-637d987c53fd_isr_898799_45.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/9165cfcd-7e59-44bb-8cb5-637d987c53fd_isr_898799_45.stderr 2024-03-13 09:46:15 proc_lsst.multi:170 [INFO] [multi] job_id 17010125 proc_lsst.multi INFO: [multi] job_id 17010125 2024-03-13 09:46:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 09:46:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010125 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010125 parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 17010125 parsl.executors.status_handling INFO: Allocated block ID 70 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70 2024-03-13 09:46:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 2024-03-13 09:46:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 09:46:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cadc4829f0a8ca5ddb0f968ad63a4d0 with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash f3524b8329f418a72513a29067bb3f64 parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 479 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/34fbd89e-e3c2-4f7f-8434-95250e671236_characterizeImage_898696_12.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/34fbd89e-e3c2-4f7f-8434-95250e671236_characterizeImage_898696_12.stderr parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/a42f0707-6139-4f6a-8c0b-fefd8e178ea6_isr_898696_12.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/a42f0707-6139-4f6a-8c0b-fefd8e178ea6_isr_898696_12.stderr 2024-03-13 09:46:16 proc_lsst.multi:170 [INFO] [multi] job_id 17010126 proc_lsst.multi INFO: [multi] job_id 17010126 2024-03-13 09:46:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 09:46:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010126 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010126 parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 17010126 parsl.executors.status_handling INFO: Allocated block ID 71 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71 2024-03-13 09:46:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 2024-03-13 09:46:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 09:46:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcacf20f1e0b71c150f1b558924b8a21 with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash 5f8b1ca3d1f9f8ad7f99f751c7767895 parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6605e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 480 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/38f5c59d-3926-42a3-9703-846d89729d50_characterizeImage_898694_56.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/38f5c59d-3926-42a3-9703-846d89729d50_characterizeImage_898694_56.stderr parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/8a4774b5-bb47-4531-be36-312ee00da89f_isr_898694_56.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/8a4774b5-bb47-4531-be36-312ee00da89f_isr_898694_56.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85aad82836761243bba754bff6dc2907 with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash 04e4e9e9c4c0412e7c1fa4100dc58a67 parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6602c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 481 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/d8a79d9c-9506-44f5-9fb4-e1cc35c05e01_characterizeImage_898695_12.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/d8a79d9c-9506-44f5-9fb4-e1cc35c05e01_characterizeImage_898695_12.stderr parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/6fc7ee91-245c-4583-8cc2-a4670a35ce15_isr_898695_12.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/6fc7ee91-245c-4583-8cc2-a4670a35ce15_isr_898695_12.stderr parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3ac1e4e08c2668a003fdd36a4ea1a51 with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 3948a3fa90e57122ce004d94151db0ac parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 482 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/706376c1-bd97-4836-876f-f87627b3fbb5_characterizeImage_898695_54.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/706376c1-bd97-4836-876f-f87627b3fbb5_characterizeImage_898695_54.stderr parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/d8f5536f-1a7a-43cb-b646-d6115cffa3ae_isr_898695_54.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/d8f5536f-1a7a-43cb-b646-d6115cffa3ae_isr_898695_54.stderr 2024-03-13 09:46:17 proc_lsst.multi:170 [INFO] [multi] job_id 17010127 proc_lsst.multi INFO: [multi] job_id 17010127 2024-03-13 09:46:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 09:46:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010127 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010127 parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 17010127 parsl.executors.status_handling INFO: Allocated block ID 72 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72 2024-03-13 09:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 2024-03-13 09:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 09:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ff285a57d2e80457f29109db302582b with result from task 390 parsl.dataflow.dflow DEBUG: Task 390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash 803724b20eb447b5fee3f20f596264d3 parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d662840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 483 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/3143bd25-3fe9-4373-ad0c-e4d787503f78_characterizeImage_898695_2.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/3143bd25-3fe9-4373-ad0c-e4d787503f78_characterizeImage_898695_2.stderr parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/c5d69f9a-890a-4e9d-a53d-7c6ad6bfcd72_isr_898695_2.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/c5d69f9a-890a-4e9d-a53d-7c6ad6bfcd72_isr_898695_2.stderr parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3146aea599e59e9b9f772bde42bf2834 with result from task 375 parsl.dataflow.dflow DEBUG: Task 375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 376 has memoization hash 90057db75adb2adb25cc4560cd4cf09a parsl.dataflow.memoization INFO: Task 376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6618a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 484 parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/485c3048-52b1-49fb-b370-d22677c14191_characterizeImage_898694_40.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/485c3048-52b1-49fb-b370-d22677c14191_characterizeImage_898694_40.stderr parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/2d2f834d-e885-4b61-b86c-5d951d0ddf60_isr_898694_40.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/2d2f834d-e885-4b61-b86c-5d951d0ddf60_isr_898694_40.stderr parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25daa97b3a9e5535b9116128119d8a86 with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash b74f48c3411c403d315fbbccf354565d parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d660f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 485 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/5bf44745-612d-4575-bc47-64996e12fe68_characterizeImage_898695_40.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/5bf44745-612d-4575-bc47-64996e12fe68_characterizeImage_898695_40.stderr parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/57cd5305-9d70-4b75-afe5-05e223cd36a4_isr_898695_40.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/57cd5305-9d70-4b75-afe5-05e223cd36a4_isr_898695_40.stderr parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d8fad8aba8f43075bb8798e8eba9dae with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 34d44fe7d331f8695ba08a20628c97cd parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d660c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 486 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/28b92bf0-1493-4b2a-bf77-8b7085c9a03a_characterizeImage_898695_49.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/28b92bf0-1493-4b2a-bf77-8b7085c9a03a_characterizeImage_898695_49.stderr parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/07b4d63b-d3c1-4275-959d-ed59e46e2eb5_isr_898695_49.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/07b4d63b-d3c1-4275-959d-ed59e46e2eb5_isr_898695_49.stderr 2024-03-13 09:46:19 proc_lsst.multi:170 [INFO] [multi] job_id 17010128 proc_lsst.multi INFO: [multi] job_id 17010128 2024-03-13 09:46:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 09:46:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010128 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010128 parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 17010128 parsl.executors.status_handling INFO: Allocated block ID 73 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73 2024-03-13 09:46:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 2024-03-13 09:46:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 09:46:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c28243152475bd4ed482dc825c9c6649 with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash c9f825e3b1483d32f51c22c3c93e7586 parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 487 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/b047e463-5d61-4e98-9862-f6a0706baec7_characterizeImage_898799_15.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/b047e463-5d61-4e98-9862-f6a0706baec7_characterizeImage_898799_15.stderr parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/4ab4ad42-0553-4adb-af27-fe36f2df2355_isr_898799_15.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/4ab4ad42-0553-4adb-af27-fe36f2df2355_isr_898799_15.stderr parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b5b4157db5f714fcadf66d4420c5e27 with result from task 372 parsl.dataflow.dflow DEBUG: Task 372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 373 has memoization hash bdcdc25d93577d945ab35ba388319ed4 parsl.dataflow.memoization INFO: Task 373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d661580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 488 parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/a1772282-ee4f-4e83-83af-cebb66455c00_characterizeImage_898696_43.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/a1772282-ee4f-4e83-83af-cebb66455c00_characterizeImage_898696_43.stderr parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/974283d0-2718-4039-b583-a4af09f46084_isr_898696_43.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/974283d0-2718-4039-b583-a4af09f46084_isr_898696_43.stderr parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edee5859b706c3a347e460adf71a5b51 with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 4c2b19577eb89e0c3554fba8314925b7 parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d661260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 489 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/5463ca1b-cad7-4ff8-a86c-3c0f15066055_characterizeImage_898696_38.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/5463ca1b-cad7-4ff8-a86c-3c0f15066055_characterizeImage_898696_38.stderr parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/f81c8bab-1c17-4627-908f-994ca20e90fa_isr_898696_38.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/f81c8bab-1c17-4627-908f-994ca20e90fa_isr_898696_38.stderr 2024-03-13 09:46:20 proc_lsst.multi:170 [INFO] [multi] job_id 17010129 proc_lsst.multi INFO: [multi] job_id 17010129 2024-03-13 09:46:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 09:46:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010129 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010129 parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 17010129 parsl.executors.status_handling INFO: Allocated block ID 74 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74 2024-03-13 09:46:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 2024-03-13 09:46:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 09:46:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 276e40f201ff15fb4e6724fbc3d5568e with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 6b972da7e648f8dcf849e4934297a7e6 parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 490 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/abf42ef6-80ee-4ac7-873b-b2f6e1c362c9_characterizeImage_898799_27.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/abf42ef6-80ee-4ac7-873b-b2f6e1c362c9_characterizeImage_898799_27.stderr parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/d18a9ac7-9b0d-4dda-8e82-3915f972633c_isr_898799_27.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/d18a9ac7-9b0d-4dda-8e82-3915f972633c_isr_898799_27.stderr 2024-03-13 09:46:21 proc_lsst.multi:170 [INFO] [multi] job_id 17010130 proc_lsst.multi INFO: [multi] job_id 17010130 2024-03-13 09:46:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 09:46:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010130 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010130 parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 17010130 parsl.executors.status_handling INFO: Allocated block ID 75 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75 2024-03-13 09:46:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 2024-03-13 09:46:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 09:46:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb12dbf184f0254bda64a035f8153d99 with result from task 429 parsl.dataflow.dflow DEBUG: Task 429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 430 has memoization hash 2414c8dd8732e329ecc1721b1a32e621 parsl.dataflow.memoization INFO: Task 430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c481120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 491 parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/532f5d16-2fa6-4d98-b93e-c8eafefca919_characterizeImage_898799_33.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/532f5d16-2fa6-4d98-b93e-c8eafefca919_characterizeImage_898799_33.stderr parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/56f892c7-d61d-4ecd-b035-8fce4a816444_isr_898799_33.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/56f892c7-d61d-4ecd-b035-8fce4a816444_isr_898799_33.stderr 2024-03-13 09:46:23 proc_lsst.multi:170 [INFO] [multi] job_id 17010131 proc_lsst.multi INFO: [multi] job_id 17010131 2024-03-13 09:46:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010131 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010131 parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 17010131 parsl.executors.status_handling INFO: Allocated block ID 76 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76 2024-03-13 09:46:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 2024-03-13 09:46:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4904bc3bede6e31c68934dae0d471972 with result from task 405 parsl.dataflow.dflow DEBUG: Task 405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 406 has memoization hash 073d7ad39894d8a0e58d4c3e1ac32acf parsl.dataflow.memoization INFO: Task 406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6639c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 492 parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/21c9b60c-a0a4-4709-a8f0-f5de8397c2e9_characterizeImage_898800_25.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/21c9b60c-a0a4-4709-a8f0-f5de8397c2e9_characterizeImage_898800_25.stderr parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/2e45e633-ff91-49a1-81a4-949b01651591_isr_898800_25.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/2e45e633-ff91-49a1-81a4-949b01651591_isr_898800_25.stderr parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 481d6e7ff72b6780a438b755d4a10d98 with result from task 396 parsl.dataflow.dflow DEBUG: Task 396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 397 has memoization hash 9295d614d0ade23590ffe8702a1715a9 parsl.dataflow.memoization INFO: Task 397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d662e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 493 parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/3037ca79-76da-4e6c-9011-dde6c3de88ae_characterizeImage_898694_7.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/3037ca79-76da-4e6c-9011-dde6c3de88ae_characterizeImage_898694_7.stderr parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/b05ec6ba-2e4e-4fec-b5b3-892e79708b11_isr_898694_7.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/b05ec6ba-2e4e-4fec-b5b3-892e79708b11_isr_898694_7.stderr 2024-03-13 09:46:24 proc_lsst.multi:170 [INFO] [multi] job_id 17010132 proc_lsst.multi INFO: [multi] job_id 17010132 2024-03-13 09:46:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-13 09:46:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010132 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010132 parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 17010132 parsl.executors.status_handling INFO: Allocated block ID 77 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77 2024-03-13 09:46:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 2024-03-13 09:46:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-13 09:46:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13dc107ce667610dfb0153b285e05bb0 with result from task 432 parsl.dataflow.dflow DEBUG: Task 432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 433 has memoization hash b5000eeee80af68bf6cf9a1472c211ee parsl.dataflow.memoization INFO: Task 433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c481620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 494 parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/469ce836-8c34-47f8-afc5-72c34a62505c_characterizeImage_898800_53.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/469ce836-8c34-47f8-afc5-72c34a62505c_characterizeImage_898800_53.stderr parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/f68b3a1f-661f-4c43-b3dd-00cc3d0e5073_isr_898800_53.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/f68b3a1f-661f-4c43-b3dd-00cc3d0e5073_isr_898800_53.stderr parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c101dbc58d13590a319df800a5e6dcea with result from task 408 parsl.dataflow.dflow DEBUG: Task 408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 409 has memoization hash 31768357e251dce6afe3aa277f54e222 parsl.dataflow.memoization INFO: Task 409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d663ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 495 parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/54f83227-aa48-4a4c-8ea8-c24c53810802_characterizeImage_898799_7.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/54f83227-aa48-4a4c-8ea8-c24c53810802_characterizeImage_898799_7.stderr parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/44044ec4-d51f-4ce6-8b2f-9530d082ab69_isr_898799_7.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/44044ec4-d51f-4ce6-8b2f-9530d082ab69_isr_898799_7.stderr parsl.dataflow.dflow INFO: Task 438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3a0a9682538ac7ef0a675d477618771 with result from task 438 parsl.dataflow.dflow DEBUG: Task 438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 439 has memoization hash b381f2df92226b0e4adbadb205c2fe70 parsl.dataflow.memoization INFO: Task 439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c481c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 496 parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/fdf0233a-36d7-4797-bf98-017281565e88_characterizeImage_898800_47.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/fdf0233a-36d7-4797-bf98-017281565e88_characterizeImage_898800_47.stderr parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/1bb7ce4e-3cc4-42f1-bc34-f8f68cf3db40_isr_898800_47.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/1bb7ce4e-3cc4-42f1-bc34-f8f68cf3db40_isr_898800_47.stderr parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5cbe179f08678493596a7fb7cb16665 with result from task 420 parsl.dataflow.dflow DEBUG: Task 420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 421 has memoization hash 46484aa14e27ea42b4c2a76484b74e20 parsl.dataflow.memoization INFO: Task 421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4807c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 497 parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/80292448-a1f3-4c0b-aa1c-500df8ac5d4a_characterizeImage_898798_9.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/80292448-a1f3-4c0b-aa1c-500df8ac5d4a_characterizeImage_898798_9.stderr parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/c730bd0e-2f4e-4b34-a01d-35230a9c68e6_isr_898798_9.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/c730bd0e-2f4e-4b34-a01d-35230a9c68e6_isr_898798_9.stderr 2024-03-13 09:46:25 proc_lsst.multi:170 [INFO] [multi] job_id 17010133 proc_lsst.multi INFO: [multi] job_id 17010133 2024-03-13 09:46:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-13 09:46:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010133 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010133 parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 17010133 parsl.executors.status_handling INFO: Allocated block ID 78 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78 2024-03-13 09:46:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 2024-03-13 09:46:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-13 09:46:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5aeb0134206aa8b6a39a69b9f9909ccb with result from task 426 parsl.dataflow.dflow DEBUG: Task 426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 427 has memoization hash 0e15c3429c545ad1d7c33563d0faba8c parsl.dataflow.memoization INFO: Task 427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c480e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 498 parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/af7455e0-8179-41cc-b9aa-8d7b23eace84_characterizeImage_898799_19.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/af7455e0-8179-41cc-b9aa-8d7b23eace84_characterizeImage_898799_19.stderr parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/a0caebc5-8588-489c-9d43-b55f66962db9_isr_898799_19.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/a0caebc5-8588-489c-9d43-b55f66962db9_isr_898799_19.stderr parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb80225445e8c58479117fd75703d4e7 with result from task 399 parsl.dataflow.dflow DEBUG: Task 399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 400 has memoization hash d53dd4499f3fd65a3315268deb72c954 parsl.dataflow.memoization INFO: Task 400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6631a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 499 parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/6ffe6d66-005b-4fa8-ade1-fd1a2fec53d7_characterizeImage_898694_22.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/6ffe6d66-005b-4fa8-ade1-fd1a2fec53d7_characterizeImage_898694_22.stderr parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/9ddcf504-0d9f-40c6-99b0-a322f43a1c47_isr_898694_22.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/9ddcf504-0d9f-40c6-99b0-a322f43a1c47_isr_898694_22.stderr parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ec9d46e88c25a2cfe972869ab644852 with result from task 414 parsl.dataflow.dflow DEBUG: Task 414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash 2546251b30857eab03dad54061293d7a parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6634c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 500 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/1f61806b-9e10-4b93-ab52-5f1eb8e89227_characterizeImage_898695_46.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/1f61806b-9e10-4b93-ab52-5f1eb8e89227_characterizeImage_898695_46.stderr parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/dc3d9518-93a7-4d78-b482-1c256c0526b6_isr_898695_46.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/dc3d9518-93a7-4d78-b482-1c256c0526b6_isr_898695_46.stderr 2024-03-13 09:46:26 proc_lsst.multi:170 [INFO] [multi] job_id 17010134 proc_lsst.multi INFO: [multi] job_id 17010134 2024-03-13 09:46:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-13 09:46:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010134 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010134 parsl.executors.status_handling DEBUG: Launched block 78 on executor multi with job ID 17010134 parsl.executors.status_handling INFO: Allocated block ID 79 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79 2024-03-13 09:46:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 2024-03-13 09:46:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-13 09:46:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d01fc3cb81e6db0027c946c893c340f with result from task 435 parsl.dataflow.dflow DEBUG: Task 435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 436 has memoization hash 55bf0a37e3818847ba848ee01214ea69 parsl.dataflow.memoization INFO: Task 436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c481940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 436 try 0 launched on executor multi with executor id 501 parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/20a14898-7ce5-4337-b438-9e3f712403ad_characterizeImage_898695_6.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/20a14898-7ce5-4337-b438-9e3f712403ad_characterizeImage_898695_6.stderr parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/dc05af37-e7a7-4587-af35-d1810d260e5e_isr_898695_6.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/dc05af37-e7a7-4587-af35-d1810d260e5e_isr_898695_6.stderr parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 290c2601654747cc11c06254ae98f96d with result from task 402 parsl.dataflow.dflow DEBUG: Task 402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 403 has memoization hash ea2e8e492d2c7029be923f548281db20 parsl.dataflow.memoization INFO: Task 403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d663740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 502 parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/af3964b9-6412-45b2-b6aa-8529883dd89b_characterizeImage_898800_62.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/af3964b9-6412-45b2-b6aa-8529883dd89b_characterizeImage_898800_62.stderr parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/73527873-438d-4fcc-aaff-2d2c5b18bc23_isr_898800_62.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/73527873-438d-4fcc-aaff-2d2c5b18bc23_isr_898800_62.stderr parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff4da2ae0b3c5638c2006755cd107cc4 with result from task 417 parsl.dataflow.dflow DEBUG: Task 417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 418 has memoization hash 7cbb9adf649bbfdab42fac0bb84f2b5b parsl.dataflow.memoization INFO: Task 418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c480180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 503 parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/a9d7e5f4-3e0e-4dc4-9d77-cee7195913e7_characterizeImage_898800_35.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/a9d7e5f4-3e0e-4dc4-9d77-cee7195913e7_characterizeImage_898800_35.stderr parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/c349fcb6-62e4-4242-bc76-7863a2894075_isr_898800_35.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/c349fcb6-62e4-4242-bc76-7863a2894075_isr_898800_35.stderr parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76bfee96c5f15566b72ecba9264f0b3e with result from task 393 parsl.dataflow.dflow DEBUG: Task 393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 394 has memoization hash 678a1694eac1e4d7aea22b1744629179 parsl.dataflow.memoization INFO: Task 394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d662b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 504 parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/86057240-0171-437d-8008-20f4b0fc5588_characterizeImage_898799_6.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/86057240-0171-437d-8008-20f4b0fc5588_characterizeImage_898799_6.stderr parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/7b6e8027-6a97-4b26-9939-7505471b2947_isr_898799_6.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/7b6e8027-6a97-4b26-9939-7505471b2947_isr_898799_6.stderr parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24ce2292ee99b1d8583ebcfc5246b1d9 with result from task 384 parsl.dataflow.dflow DEBUG: Task 384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 385 has memoization hash 91c4defa0234fbc0346a8ef188b282a4 parsl.dataflow.memoization INFO: Task 385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d662200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 505 parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/491098e3-1072-443d-b1f7-7553527ef2f4_characterizeImage_898800_26.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/491098e3-1072-443d-b1f7-7553527ef2f4_characterizeImage_898800_26.stderr parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/39f80a79-d290-4793-9dac-42c3c68cfde0_isr_898800_26.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/39f80a79-d290-4793-9dac-42c3c68cfde0_isr_898800_26.stderr parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5707fe8966b4317d670ad4fa7295aa3 with result from task 411 parsl.dataflow.dflow DEBUG: Task 411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 412 has memoization hash 2bc603f8bc67564026f1b756dfe2dd24 parsl.dataflow.memoization INFO: Task 412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d663e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 506 parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/89a05b45-4b42-4b4b-80ab-ea4cd55f6e7a_characterizeImage_898694_6.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/89a05b45-4b42-4b4b-80ab-ea4cd55f6e7a_characterizeImage_898694_6.stderr parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/69f402cc-202f-4315-ab84-2970966a51ea_isr_898694_6.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/69f402cc-202f-4315-ab84-2970966a51ea_isr_898694_6.stderr parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e856ddf266f963a108c733d17ef39952 with result from task 387 parsl.dataflow.dflow DEBUG: Task 387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 388 has memoization hash 2dfcfff2270fa7d929a30db7700cad80 parsl.dataflow.memoization INFO: Task 388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d662520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 507 parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/27010e38-6ed4-4b48-ad4f-ccb5567b8c9c_characterizeImage_898694_38.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/27010e38-6ed4-4b48-ad4f-ccb5567b8c9c_characterizeImage_898694_38.stderr parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/5e6cf434-aed2-485f-904f-e78763034f7c_isr_898694_38.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/5e6cf434-aed2-485f-904f-e78763034f7c_isr_898694_38.stderr parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ae000729cc0b8a58607e1caf114bf38 with result from task 378 parsl.dataflow.dflow DEBUG: Task 378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash 6788856663694cdf56309b93a33c4541 parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d661bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 508 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/2fd47b84-d7f3-402d-a83c-fe069932c762_characterizeImage_898798_38.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/2fd47b84-d7f3-402d-a83c-fe069932c762_characterizeImage_898798_38.stderr parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/3bdd9e22-f8b8-4245-9207-9a750c42f83e_isr_898798_38.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/3bdd9e22-f8b8-4245-9207-9a750c42f83e_isr_898798_38.stderr parsl.dataflow.dflow INFO: Task 477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b850816268ec2ac94bad68ec52558fbf with result from task 477 parsl.dataflow.dflow DEBUG: Task 477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 478 has memoization hash b5eba159c48b877decafcd6876bde76f parsl.dataflow.memoization INFO: Task 478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c482e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 509 parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/54651e0d-ed4a-4b9c-b1f0-273dedbbc000_characterizeImage_898798_28.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/54651e0d-ed4a-4b9c-b1f0-273dedbbc000_characterizeImage_898798_28.stderr parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/6a611205-4a58-4bfa-8d87-37275ece9b31_isr_898798_28.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/6a611205-4a58-4bfa-8d87-37275ece9b31_isr_898798_28.stderr 2024-03-13 09:46:27 proc_lsst.multi:170 [INFO] [multi] job_id 17010135 proc_lsst.multi INFO: [multi] job_id 17010135 2024-03-13 09:46:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-13 09:46:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010135 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010135 parsl.executors.status_handling DEBUG: Launched block 79 on executor multi with job ID 17010135 parsl.executors.status_handling INFO: Allocated block ID 80 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80 2024-03-13 09:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 2024-03-13 09:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-13 09:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b23cefe75737cfbdf1d8274e65f763ab with result from task 423 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 424 has memoization hash 2f415fa9b646e456a95bea7aed006a98 parsl.dataflow.memoization INFO: Task 424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c480ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 510 parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ce197a9b-543e-445b-9a12-63fe75b05d8e_characterizeImage_898798_30.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ce197a9b-543e-445b-9a12-63fe75b05d8e_characterizeImage_898798_30.stderr parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/4d4f9af6-8be5-457a-8214-3d0903834ed6_isr_898798_30.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/4d4f9af6-8be5-457a-8214-3d0903834ed6_isr_898798_30.stderr parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5f7dd8b87554d8a96252de4a6aeae9d with result from task 381 parsl.dataflow.dflow DEBUG: Task 381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 382 has memoization hash 8a412a3eba52b2afe3afe08a7c995c9e parsl.dataflow.memoization INFO: Task 382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d661ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 511 parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/442f0996-5a0c-4f0d-b72f-c1a0f0573954_characterizeImage_898800_59.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/442f0996-5a0c-4f0d-b72f-c1a0f0573954_characterizeImage_898800_59.stderr parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/4c59969a-cf44-4d2e-bf58-4e12eb4910fa_isr_898800_59.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/4c59969a-cf44-4d2e-bf58-4e12eb4910fa_isr_898800_59.stderr parsl.dataflow.dflow INFO: Task 483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f85bfbf20abf7844941c5369e093d352 with result from task 483 parsl.dataflow.dflow DEBUG: Task 483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 484 has memoization hash 0032946f3a2d3083b1ae655053c83a0c parsl.dataflow.memoization INFO: Task 484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c483380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 512 parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/558168ff-287a-471e-9027-1bce897f750f_characterizeImage_898696_31.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/558168ff-287a-471e-9027-1bce897f750f_characterizeImage_898696_31.stderr parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/fe553a02-202f-4e5f-96ed-4cf527f4e7ef_isr_898696_31.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/fe553a02-202f-4e5f-96ed-4cf527f4e7ef_isr_898696_31.stderr parsl.dataflow.dflow INFO: Task 441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 235532de60e2ec785d6f78141e23a097 with result from task 441 parsl.dataflow.dflow DEBUG: Task 441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 442 has memoization hash 0c826b0f2e6ba964cbd7ef1a3f77a2a9 parsl.dataflow.memoization INFO: Task 442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c481f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 513 parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/98045237-eff2-4650-97d8-0f37ca8a5a52_characterizeImage_898798_50.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/98045237-eff2-4650-97d8-0f37ca8a5a52_characterizeImage_898798_50.stderr parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/7431907f-465a-40ba-b6a1-cbe288898ce3_isr_898798_50.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/7431907f-465a-40ba-b6a1-cbe288898ce3_isr_898798_50.stderr parsl.dataflow.dflow INFO: Task 456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd8a8849d14ddee7d0248186598cfe4e with result from task 456 parsl.dataflow.dflow DEBUG: Task 456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 457 has memoization hash 1378e62d0a8735467f133c64ddf613fc parsl.dataflow.memoization INFO: Task 457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 514 parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/b06bb2c5-1ca2-44e3-ba6d-40832489814f_characterizeImage_898798_2.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/b06bb2c5-1ca2-44e3-ba6d-40832489814f_characterizeImage_898798_2.stderr parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/928b569d-efbe-492e-84db-9c6aca2fd956_isr_898798_2.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/928b569d-efbe-492e-84db-9c6aca2fd956_isr_898798_2.stderr 2024-03-13 09:46:29 proc_lsst.multi:170 [INFO] [multi] job_id 17010136 proc_lsst.multi INFO: [multi] job_id 17010136 2024-03-13 09:46:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-13 09:46:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010136 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010136 parsl.executors.status_handling DEBUG: Launched block 80 on executor multi with job ID 17010136 parsl.executors.status_handling INFO: Allocated block ID 81 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81 2024-03-13 09:46:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 2024-03-13 09:46:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-13 09:46:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2d3f3d5dba1dfb3b4962f3d10032c94 with result from task 480 parsl.dataflow.dflow DEBUG: Task 480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 481 has memoization hash 6e9f44d14cdb055df20921d790fa5131 parsl.dataflow.memoization INFO: Task 481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c483060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 515 parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/babf7d0c-1098-4c39-88ef-cd5f18f1cb48_characterizeImage_898799_59.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/babf7d0c-1098-4c39-88ef-cd5f18f1cb48_characterizeImage_898799_59.stderr parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/3b960306-2e34-492a-af89-c331228ea9be_isr_898799_59.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/3b960306-2e34-492a-af89-c331228ea9be_isr_898799_59.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c8d40c29dba8c1f4e054cdb955bdbf5 with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 6a649392de091357d694f22746804dfe parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 516 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/e837ed91-5bf2-41d7-be2f-18e8220b17be_characterizeImage_898695_26.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/e837ed91-5bf2-41d7-be2f-18e8220b17be_characterizeImage_898695_26.stderr parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/6b618ae3-431b-4d5d-a65b-1870efd6b707_isr_898695_26.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/6b618ae3-431b-4d5d-a65b-1870efd6b707_isr_898695_26.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df74e0df9cf77f825e762809937592cc with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash a12f3c08fe4c8cbf0c07c30219508502 parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 517 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/1f323fe0-ea9a-45aa-8fa5-57485168192d_characterizeImage_898694_15.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/1f323fe0-ea9a-45aa-8fa5-57485168192d_characterizeImage_898694_15.stderr parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/68f93f20-aa37-4764-9c4e-0592282a4efa_isr_898694_15.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/68f93f20-aa37-4764-9c4e-0592282a4efa_isr_898694_15.stderr parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 419e2c93b19d43c2c18aef5d957f5eb2 with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash d5380e456df56f87db89b9b9b7b6963d parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 518 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/266d3074-4412-48b3-a0d7-3114d0f40f3f_characterizeImage_898695_19.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/266d3074-4412-48b3-a0d7-3114d0f40f3f_characterizeImage_898695_19.stderr parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/c0a0c7b1-1ccf-405d-9d32-7b92b534c6f0_isr_898695_19.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/c0a0c7b1-1ccf-405d-9d32-7b92b534c6f0_isr_898695_19.stderr parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0934989a531a37b186e08a957b8871bf with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 9e4bb11b3f96392e583635adf44c82e4 parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 519 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/db1c13f1-cdff-4920-a37c-97dcc9c56500_characterizeImage_898694_1.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/db1c13f1-cdff-4920-a37c-97dcc9c56500_characterizeImage_898694_1.stderr parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/5ef55cb0-dc80-47aa-a1a6-7f34a2a60495_isr_898694_1.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/5ef55cb0-dc80-47aa-a1a6-7f34a2a60495_isr_898694_1.stderr parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bdb190fd389956c27a7763eea64ee55 with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash b11ea466329b3d034fdfe08682ae16d7 parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 520 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/16f58476-05f4-465f-81b2-2d2fc3a3d462_characterizeImage_898694_25.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/16f58476-05f4-465f-81b2-2d2fc3a3d462_characterizeImage_898694_25.stderr parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/76a793c9-c574-4fc2-92ff-2d09bca463ec_isr_898694_25.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/76a793c9-c574-4fc2-92ff-2d09bca463ec_isr_898694_25.stderr parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8075a42f908e26fc924f721726568f54 with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 7bdc683351e9ff8b51ca33427b1f39f2 parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 521 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/ce7e3124-805e-4a57-851d-44abc875c31e_characterizeImage_898799_5.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/ce7e3124-805e-4a57-851d-44abc875c31e_characterizeImage_898799_5.stderr parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/410e4da0-7f12-49bd-8b23-9633dda9e433_isr_898799_5.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/410e4da0-7f12-49bd-8b23-9633dda9e433_isr_898799_5.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c757da3a19e7c564c8f561a9af0ced26 with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash cd0252bb75c256be58e402cddc270257 parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 522 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/005478be-e3b3-4992-8f7a-555c13b81168_characterizeImage_898798_17.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/005478be-e3b3-4992-8f7a-555c13b81168_characterizeImage_898798_17.stderr parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/b58144c6-66dd-4ff9-bfcf-d4e5c07afa0a_isr_898798_17.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/b58144c6-66dd-4ff9-bfcf-d4e5c07afa0a_isr_898798_17.stderr parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13a8336aba97555881b3bb5f6e1a7b1b with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash aef49d492db63ed87c2761e2d5fdcc03 parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d660900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 523 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/c3073a09-79de-4a71-9f5e-a203f0a3768a_characterizeImage_898695_27.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/c3073a09-79de-4a71-9f5e-a203f0a3768a_characterizeImage_898695_27.stderr parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/b3bb910a-acea-4dca-aa77-31f5eb7f7bce_isr_898695_27.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/b3bb910a-acea-4dca-aa77-31f5eb7f7bce_isr_898695_27.stderr parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b07cca265dfa98f724abcbe5f6ce6521 with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash c276a4ba8598a2aa837d56eb7c7e98d3 parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 524 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/cd5ab4bd-0725-4481-b89c-aa2b79a1fcfe_characterizeImage_898696_28.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/cd5ab4bd-0725-4481-b89c-aa2b79a1fcfe_characterizeImage_898696_28.stderr parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/99526a73-6d1e-4121-a6da-96aea938d8ff_isr_898696_28.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/99526a73-6d1e-4121-a6da-96aea938d8ff_isr_898696_28.stderr parsl.dataflow.dflow INFO: Task 468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 028fa278170022c309c106f41fb68115 with result from task 468 parsl.dataflow.dflow DEBUG: Task 468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 469 has memoization hash cfad0a91d41de9b78146f8fb5c51dc72 parsl.dataflow.memoization INFO: Task 469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c482160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 525 parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/51813a66-c61b-41af-8bf3-5f69d5bf333a_characterizeImage_898798_31.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/51813a66-c61b-41af-8bf3-5f69d5bf333a_characterizeImage_898798_31.stderr parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/816ab556-0529-4a57-9c60-f70b986a281f_isr_898798_31.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/816ab556-0529-4a57-9c60-f70b986a281f_isr_898798_31.stderr parsl.dataflow.dflow INFO: Task 462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93547fb9205d888c0b29e35336f521d6 with result from task 462 parsl.dataflow.dflow DEBUG: Task 462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 463 has memoization hash de9e54deccef1d8d46dc2a788b32138f parsl.dataflow.memoization INFO: Task 463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 526 parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/727ab573-8ad1-437d-94f3-a33f49b25ff6_characterizeImage_898798_56.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/727ab573-8ad1-437d-94f3-a33f49b25ff6_characterizeImage_898798_56.stderr parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/342b8674-1c22-46c5-9afe-0c82f2040cb9_isr_898798_56.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/342b8674-1c22-46c5-9afe-0c82f2040cb9_isr_898798_56.stderr parsl.dataflow.dflow INFO: Task 459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1f8a95511dd431112c0341194fcc62d with result from task 459 parsl.dataflow.dflow DEBUG: Task 459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 460 has memoization hash e0c312e47bb02479fe6f6e20d1982f7a parsl.dataflow.memoization INFO: Task 460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 527 parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/637a89f5-32e4-4123-a566-533fa29846ec_characterizeImage_898695_11.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/637a89f5-32e4-4123-a566-533fa29846ec_characterizeImage_898695_11.stderr parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/86ea0176-2a82-4723-bd29-bb72d84589b2_isr_898695_11.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/86ea0176-2a82-4723-bd29-bb72d84589b2_isr_898695_11.stderr parsl.dataflow.dflow INFO: Task 471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 362ae681671355166b4a95eeb624471c with result from task 471 2024-03-13 09:46:30 proc_lsst.multi:170 [INFO] [multi] job_id 17010137 proc_lsst.multi INFO: [multi] job_id 17010137 2024-03-13 09:46:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-13 09:46:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010137 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010137 parsl.executors.status_handling DEBUG: Launched block 81 on executor multi with job ID 17010137 parsl.executors.status_handling INFO: Allocated block ID 82 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82 2024-03-13 09:46:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 2024-03-13 09:46:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-13 09:46:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 472 has memoization hash 8c51004598d45827d1a4f9a387e039a1 parsl.dataflow.memoization INFO: Task 472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4827a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 528 parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/f9015afc-4f8f-4f6a-9e0d-944dd4b89f5b_characterizeImage_898695_14.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/f9015afc-4f8f-4f6a-9e0d-944dd4b89f5b_characterizeImage_898695_14.stderr parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/ceb28092-d6c7-40b8-82e8-1bb1426fb50b_isr_898695_14.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/ceb28092-d6c7-40b8-82e8-1bb1426fb50b_isr_898695_14.stderr parsl.dataflow.dflow INFO: Task 465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d4944062fa6c12bfa978ed1b1ac38d0 with result from task 465 parsl.dataflow.dflow DEBUG: Task 465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 466 has memoization hash dd4d47c66a8dfd3660f44015c96bc6ec parsl.dataflow.memoization INFO: Task 466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 529 parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/176ae148-7e63-4550-ab80-ae100c2bdf2f_characterizeImage_898695_10.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/176ae148-7e63-4550-ab80-ae100c2bdf2f_characterizeImage_898695_10.stderr parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/535e504c-a5b5-417b-abfa-7f8b72b9af02_isr_898695_10.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/535e504c-a5b5-417b-abfa-7f8b72b9af02_isr_898695_10.stderr parsl.dataflow.dflow INFO: Task 444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb29a272368cfb93323e5542cb3e6db6 with result from task 444 parsl.dataflow.dflow DEBUG: Task 444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 445 has memoization hash d8379fd5146b74ab9ce34d04e57767f6 parsl.dataflow.memoization INFO: Task 445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 530 parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/07726702-b872-4f1e-8caa-f70abd574a66_characterizeImage_898798_55.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/07726702-b872-4f1e-8caa-f70abd574a66_characterizeImage_898798_55.stderr parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/fc8a0b31-dbd8-4709-8ecb-bfed76ad324c_isr_898798_55.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/fc8a0b31-dbd8-4709-8ecb-bfed76ad324c_isr_898798_55.stderr parsl.dataflow.dflow INFO: Task 486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e6b00a5d1b4a8e9793ef3a183ecc2fd with result from task 486 parsl.dataflow.dflow DEBUG: Task 486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 487 has memoization hash bcc595aa24df6fa0a580b03d8d9048be parsl.dataflow.memoization INFO: Task 487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4836a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 531 parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/6917b83a-7673-43f6-9ccf-bb14004f3dee_characterizeImage_898694_27.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/6917b83a-7673-43f6-9ccf-bb14004f3dee_characterizeImage_898694_27.stderr parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/80ce196e-a08f-447d-943d-1cc22b1fef20_isr_898694_27.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/80ce196e-a08f-447d-943d-1cc22b1fef20_isr_898694_27.stderr parsl.dataflow.dflow INFO: Task 447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7693b413523962712cffd017f7a548bd with result from task 447 parsl.dataflow.dflow DEBUG: Task 447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 448 has memoization hash 4d75758b6433e05d1ab259217bc2cde4 parsl.dataflow.memoization INFO: Task 448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 532 parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/40fe8503-3ff8-4b5d-b5e1-a9455b6083fa_characterizeImage_898695_43.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/40fe8503-3ff8-4b5d-b5e1-a9455b6083fa_characterizeImage_898695_43.stderr parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/041488c2-7bd7-4816-9036-d51ab89cb225_isr_898695_43.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/041488c2-7bd7-4816-9036-d51ab89cb225_isr_898695_43.stderr parsl.dataflow.dflow INFO: Task 474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c927f7c3fbaa047d229d30ac94cf305a with result from task 474 parsl.dataflow.dflow DEBUG: Task 474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 475 has memoization hash 7095f1c19f0176a35bf899f268c6b121 parsl.dataflow.memoization INFO: Task 475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c482b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 533 parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/b7fb183b-a3ef-49a7-a0d2-ade2e10eb4ca_characterizeImage_898800_48.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/b7fb183b-a3ef-49a7-a0d2-ade2e10eb4ca_characterizeImage_898800_48.stderr parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/0d9feac5-23e7-4611-966b-0045d66b5282_isr_898800_48.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/0d9feac5-23e7-4611-966b-0045d66b5282_isr_898800_48.stderr parsl.dataflow.dflow INFO: Task 495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95ce67f6b0938705d23a89dc4da57b56 with result from task 495 parsl.dataflow.dflow DEBUG: Task 495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 496 has memoization hash b30ca7ee3c1ea56b99d8bc84d0453cc1 parsl.dataflow.memoization INFO: Task 496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c483ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 534 parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/e9b3f818-f495-4062-8a52-442a54674f59_characterizeImage_898695_8.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/e9b3f818-f495-4062-8a52-442a54674f59_characterizeImage_898695_8.stderr parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/dd8d0a39-f831-43ee-ae23-317f60f26b98_isr_898695_8.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/dd8d0a39-f831-43ee-ae23-317f60f26b98_isr_898695_8.stderr parsl.dataflow.dflow INFO: Task 501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0627e575398cd5ea2a65c7ca1f4d95a9 with result from task 501 parsl.dataflow.dflow DEBUG: Task 501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 502 has memoization hash 700562c63a7f19bd1624ed7627dc6aa9 parsl.dataflow.memoization INFO: Task 502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 09:46:31 proc_lsst.multi:170 [INFO] [multi] job_id 17010139 proc_lsst.multi INFO: [multi] job_id 17010139 2024-03-13 09:46:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-13 09:46:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010139 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010139 parsl.executors.status_handling DEBUG: Launched block 82 on executor multi with job ID 17010139 parsl.executors.status_handling INFO: Allocated block ID 83 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83 2024-03-13 09:46:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 2024-03-13 09:46:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-13 09:46:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 535 parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/75f5eb78-e542-4c2a-8696-47d44f63c5e1_characterizeImage_898798_58.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/75f5eb78-e542-4c2a-8696-47d44f63c5e1_characterizeImage_898798_58.stderr parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/aee5f342-a869-4dfb-a1a2-f4d0a95a5b51_isr_898798_58.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/aee5f342-a869-4dfb-a1a2-f4d0a95a5b51_isr_898798_58.stderr parsl.dataflow.dflow INFO: Task 489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7ce022246a5ab6230a9fa7911efc92d with result from task 489 parsl.dataflow.dflow DEBUG: Task 489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 490 has memoization hash cf65f16682d1373f3e771f0570906e9f parsl.dataflow.memoization INFO: Task 490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c483b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 536 parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ce6b58e3-4ded-415f-9235-bdd55175c8c8_characterizeImage_898798_62.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ce6b58e3-4ded-415f-9235-bdd55175c8c8_characterizeImage_898798_62.stderr parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/aaa86e50-3c95-4512-9778-9b7ddd1caa73_isr_898798_62.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/aaa86e50-3c95-4512-9778-9b7ddd1caa73_isr_898798_62.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e39e3f1de2f244c4c38ed4b3fbed4a4 with result from task 492 parsl.dataflow.dflow DEBUG: Task 492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 493 has memoization hash 18b5e8fb531024c205678fef3cf5a45c parsl.dataflow.memoization INFO: Task 493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c483e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 537 parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/de91f86e-33db-4d6d-aa34-c0829536fa8e_characterizeImage_898696_30.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/de91f86e-33db-4d6d-aa34-c0829536fa8e_characterizeImage_898696_30.stderr parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/55960bf9-aeb3-406f-a9fd-f01f0ec8c7a4_isr_898696_30.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/55960bf9-aeb3-406f-a9fd-f01f0ec8c7a4_isr_898696_30.stderr parsl.dataflow.dflow INFO: Task 453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5dfb31a11ad58dddfab0c20fc9c1241b with result from task 453 parsl.dataflow.dflow DEBUG: Task 453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 454 has memoization hash 2d7de60cf1170360143eea50b8708ba0 parsl.dataflow.memoization INFO: Task 454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 538 parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/c1a5c801-fc27-4796-b92e-fe15017c2969_characterizeImage_898696_48.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/c1a5c801-fc27-4796-b92e-fe15017c2969_characterizeImage_898696_48.stderr parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/c291f742-589b-4187-9f84-070ee3cda67f_isr_898696_48.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/c291f742-589b-4187-9f84-070ee3cda67f_isr_898696_48.stderr 2024-03-13 09:46:33 proc_lsst.multi:170 [INFO] [multi] job_id 17010140 proc_lsst.multi INFO: [multi] job_id 17010140 2024-03-13 09:46:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-13 09:46:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010140 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010140 parsl.executors.status_handling DEBUG: Launched block 83 on executor multi with job ID 17010140 parsl.executors.status_handling INFO: Allocated block ID 84 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84 2024-03-13 09:46:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 2024-03-13 09:46:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-13 09:46:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b78c98ca00d601fb53163b1b63b4b39 with result from task 507 parsl.dataflow.dflow DEBUG: Task 507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 508 has memoization hash 565ce4012fd324c4d9a16e0f3513f25d parsl.dataflow.memoization INFO: Task 508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 539 parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/1d2f74a7-7c94-4755-8a1a-8d084c00712b_characterizeImage_898799_39.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/1d2f74a7-7c94-4755-8a1a-8d084c00712b_characterizeImage_898799_39.stderr parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/ca85b22e-7b93-4e92-ab7c-2e2c5794ce98_isr_898799_39.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/ca85b22e-7b93-4e92-ab7c-2e2c5794ce98_isr_898799_39.stderr parsl.dataflow.dflow INFO: Task 450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c399632e4d15fc60e44f6774315947f6 with result from task 450 parsl.dataflow.dflow DEBUG: Task 450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 451 has memoization hash 858b000de5eaec7e698e1d7207d49da8 parsl.dataflow.memoization INFO: Task 451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 540 parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/ab9f05ca-98db-4b35-ab6d-25174f68d32e_characterizeImage_898799_60.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/ab9f05ca-98db-4b35-ab6d-25174f68d32e_characterizeImage_898799_60.stderr parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/ea236db1-82e1-43b0-a35a-827c2848e522_isr_898799_60.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/ea236db1-82e1-43b0-a35a-827c2848e522_isr_898799_60.stderr 2024-03-13 09:46:34 proc_lsst.multi:170 [INFO] [multi] job_id 17010141 proc_lsst.multi INFO: [multi] job_id 17010141 2024-03-13 09:46:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-13 09:46:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010141 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010141 parsl.executors.status_handling DEBUG: Launched block 84 on executor multi with job ID 17010141 parsl.executors.status_handling INFO: Allocated block ID 85 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85 2024-03-13 09:46:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 2024-03-13 09:46:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-13 09:46:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 612c4b65bf79b92b6e00a0bc98e3dc23 with result from task 510 parsl.dataflow.dflow DEBUG: Task 510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 511 has memoization hash 92ae2f671eb39cd112f41db07d987799 parsl.dataflow.memoization INFO: Task 511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 541 parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/e7b9bf64-2407-4a77-8bf6-d09ef063c948_characterizeImage_898799_11.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/e7b9bf64-2407-4a77-8bf6-d09ef063c948_characterizeImage_898799_11.stderr parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/bfcb9197-9596-488e-b2b3-77c75a0f16bd_isr_898799_11.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/bfcb9197-9596-488e-b2b3-77c75a0f16bd_isr_898799_11.stderr parsl.dataflow.dflow INFO: Task 498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c177bfadb1212b0b09e419b82caa537 with result from task 498 parsl.dataflow.dflow DEBUG: Task 498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 499 has memoization hash 24b3544f3910716e867e86c5546a070d parsl.dataflow.memoization INFO: Task 499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 542 parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/44f4b2af-92c4-4126-b94f-068da0e4f109_characterizeImage_898799_30.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/44f4b2af-92c4-4126-b94f-068da0e4f109_characterizeImage_898799_30.stderr parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/8ef8c210-3705-41be-a20e-036e59ebcd34_isr_898799_30.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/8ef8c210-3705-41be-a20e-036e59ebcd34_isr_898799_30.stderr parsl.dataflow.dflow INFO: Task 516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a8b698c96bf84043a33540699949d12 with result from task 516 parsl.dataflow.dflow DEBUG: Task 516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 517 has memoization hash 1a423e548ce6d9f6d09379a97f6fb0e1 parsl.dataflow.memoization INFO: Task 517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 543 parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/89ec9623-551b-4040-9885-f9211f0fda54_characterizeImage_898695_56.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/89ec9623-551b-4040-9885-f9211f0fda54_characterizeImage_898695_56.stderr parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/88269940-b43f-490c-9ae7-0610e5ae2ae2_isr_898695_56.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/88269940-b43f-490c-9ae7-0610e5ae2ae2_isr_898695_56.stderr parsl.dataflow.dflow INFO: Task 519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry caf254003b429a427899f2222a8dd6e6 with result from task 519 parsl.dataflow.dflow DEBUG: Task 519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 520 has memoization hash dce3c26bb613b8350e34998c2b76c9f4 parsl.dataflow.memoization INFO: Task 520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 544 parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/6e8b5693-e5f1-48cb-af12-051a79254362_characterizeImage_898799_44.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/6e8b5693-e5f1-48cb-af12-051a79254362_characterizeImage_898799_44.stderr parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/d41c724e-4b56-41fc-8207-efe268e6fb53_isr_898799_44.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/d41c724e-4b56-41fc-8207-efe268e6fb53_isr_898799_44.stderr 2024-03-13 09:46:35 proc_lsst.multi:170 [INFO] [multi] job_id 17010142 proc_lsst.multi INFO: [multi] job_id 17010142 2024-03-13 09:46:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010142 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010142 parsl.executors.status_handling DEBUG: Launched block 85 on executor multi with job ID 17010142 parsl.executors.status_handling INFO: Allocated block ID 86 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86 2024-03-13 09:46:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 2024-03-13 09:46:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3675c1e13b007482af60e6759384eef9 with result from task 504 parsl.dataflow.dflow DEBUG: Task 504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 505 has memoization hash 8c4844f4052f89f292c94da53fee16bf parsl.dataflow.memoization INFO: Task 505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 505 try 0 launched on executor multi with executor id 545 parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/a6152900-2763-4d37-bbed-803a41b064f2_characterizeImage_898800_37.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/a6152900-2763-4d37-bbed-803a41b064f2_characterizeImage_898800_37.stderr parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/38fb58fb-7834-4883-8891-792091a6bf30_isr_898800_37.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/38fb58fb-7834-4883-8891-792091a6bf30_isr_898800_37.stderr parsl.dataflow.dflow INFO: Task 525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b923bb61c5e3ac08301708d966e6aea4 with result from task 525 parsl.dataflow.dflow DEBUG: Task 525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 526 has memoization hash 993fccca5b86d3f1c0803e8366ba11d3 parsl.dataflow.memoization INFO: Task 526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 546 parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/59196439-7766-4447-8f1c-be4e3e692698_characterizeImage_898694_17.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/59196439-7766-4447-8f1c-be4e3e692698_characterizeImage_898694_17.stderr parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/3ef4d5e6-d4fe-4d3e-a9e2-1b9eff80e16b_isr_898694_17.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/3ef4d5e6-d4fe-4d3e-a9e2-1b9eff80e16b_isr_898694_17.stderr 2024-03-13 09:46:36 proc_lsst.multi:170 [INFO] [multi] job_id 17010143 proc_lsst.multi INFO: [multi] job_id 17010143 2024-03-13 09:46:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-13 09:46:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010143 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010143 parsl.executors.status_handling DEBUG: Launched block 86 on executor multi with job ID 17010143 parsl.executors.status_handling INFO: Allocated block ID 87 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87 2024-03-13 09:46:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 2024-03-13 09:46:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-13 09:46:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17d430884a10802d5dc692449217b3ee with result from task 531 parsl.dataflow.dflow DEBUG: Task 531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 532 has memoization hash 8258377d21655f9632bedb56f4966a36 parsl.dataflow.memoization INFO: Task 532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 547 parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/c25fe78b-493a-444c-9362-81318b35066f_characterizeImage_898695_31.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/c25fe78b-493a-444c-9362-81318b35066f_characterizeImage_898695_31.stderr parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/297005f9-8874-4e08-aecd-dcc9bf20c33c_isr_898695_31.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/297005f9-8874-4e08-aecd-dcc9bf20c33c_isr_898695_31.stderr parsl.dataflow.dflow INFO: Task 612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bc40bcf919347d1af4325c61da814fe with result from task 612 parsl.dataflow.dflow DEBUG: Task 612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 613 has memoization hash bb33f33e57570615950e88d72ab24172 parsl.dataflow.memoization INFO: Task 613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c511440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 613 try 0 launched on executor multi with executor id 548 parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/f1af8fb0-c19f-4db8-9e59-14ca589c948a_characterizeImage_898800_30.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/f1af8fb0-c19f-4db8-9e59-14ca589c948a_characterizeImage_898800_30.stderr parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/95a121b3-6d6b-46d8-a76a-85cf7acef544_isr_898800_30.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/95a121b3-6d6b-46d8-a76a-85cf7acef544_isr_898800_30.stderr 2024-03-13 09:46:37 proc_lsst.multi:170 [INFO] [multi] job_id 17010144 proc_lsst.multi INFO: [multi] job_id 17010144 2024-03-13 09:46:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-13 09:46:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010144 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010144 parsl.executors.status_handling DEBUG: Launched block 87 on executor multi with job ID 17010144 parsl.executors.status_handling INFO: Allocated block ID 88 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88 2024-03-13 09:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 2024-03-13 09:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-13 09:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8daaef47930933a1692b77599fd9fb8f with result from task 528 parsl.dataflow.dflow DEBUG: Task 528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 529 has memoization hash fb8802b478b93bdcd650e1d548296994 parsl.dataflow.memoization INFO: Task 529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 549 parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/c3b350e3-39a2-4c33-ae6a-05333da96e9e_characterizeImage_898696_23.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/c3b350e3-39a2-4c33-ae6a-05333da96e9e_characterizeImage_898696_23.stderr parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/980cfec4-2043-4a92-9ddf-8dd3e22136ce_isr_898696_23.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/980cfec4-2043-4a92-9ddf-8dd3e22136ce_isr_898696_23.stderr parsl.dataflow.dflow INFO: Task 534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 102792a662ecd3f3203092e77baa9178 with result from task 534 parsl.dataflow.dflow DEBUG: Task 534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 535 has memoization hash de7aaa3378ffcf74ffdb971dee5ed00b parsl.dataflow.memoization INFO: Task 535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dc2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 550 parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/2f8cd3fd-975c-43e8-aac9-a2e02be710b8_characterizeImage_898696_29.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/2f8cd3fd-975c-43e8-aac9-a2e02be710b8_characterizeImage_898696_29.stderr parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/9c233f0e-faab-4b5e-bf6f-ce60b09f7870_isr_898696_29.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/9c233f0e-faab-4b5e-bf6f-ce60b09f7870_isr_898696_29.stderr 2024-03-13 09:46:39 proc_lsst.multi:170 [INFO] [multi] job_id 17010145 proc_lsst.multi INFO: [multi] job_id 17010145 2024-03-13 09:46:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-13 09:46:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010145 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010145 parsl.executors.status_handling DEBUG: Launched block 88 on executor multi with job ID 17010145 parsl.executors.status_handling INFO: Allocated block ID 89 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89 2024-03-13 09:46:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 2024-03-13 09:46:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-13 09:46:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f74dd0ed2c9931d1753a5e14228ede22 with result from task 708 parsl.dataflow.dflow DEBUG: Task 708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 709 has memoization hash 8e9b355ac0df3c0df951e5e36c1824de parsl.dataflow.memoization INFO: Task 709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c377880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 709 try 0 launched on executor multi with executor id 551 parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/68aeb1ba-3cd6-494d-88ca-919882b0dc15_characterizeImage_898696_57.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/68aeb1ba-3cd6-494d-88ca-919882b0dc15_characterizeImage_898696_57.stderr parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/e3feab52-af24-49e5-b42d-539da8101da9_isr_898696_57.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/e3feab52-af24-49e5-b42d-539da8101da9_isr_898696_57.stderr parsl.dataflow.dflow INFO: Task 513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e191d697a1bb542320f9e784c4cfdfb with result from task 513 parsl.dataflow.dflow DEBUG: Task 513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 514 has memoization hash bfa6e832111110eba0523c4ca978853b parsl.dataflow.memoization INFO: Task 514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 552 parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/971f994b-3f52-471c-890a-7a4badb5fdc8_characterizeImage_898696_53.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/971f994b-3f52-471c-890a-7a4badb5fdc8_characterizeImage_898696_53.stderr parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/3e3100ce-0ed8-4622-8b0d-2b8c93307240_isr_898696_53.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/3e3100ce-0ed8-4622-8b0d-2b8c93307240_isr_898696_53.stderr parsl.dataflow.dflow INFO: Task 615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed3f63875726d4391ea73b3d6aa85478 with result from task 615 parsl.dataflow.dflow DEBUG: Task 615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 616 has memoization hash 38222c296165b6da0fe0b1f493077f40 parsl.dataflow.memoization INFO: Task 616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c511760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 616 try 0 launched on executor multi with executor id 553 parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/4b7c1047-7cd8-456e-8371-2b489c534f3e_characterizeImage_898695_22.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/4b7c1047-7cd8-456e-8371-2b489c534f3e_characterizeImage_898695_22.stderr parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/c27af14e-1b9b-4c54-9036-1644838df165_isr_898695_22.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/c27af14e-1b9b-4c54-9036-1644838df165_isr_898695_22.stderr 2024-03-13 09:46:40 proc_lsst.multi:170 [INFO] [multi] job_id 17010146 proc_lsst.multi INFO: [multi] job_id 17010146 2024-03-13 09:46:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-13 09:46:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010146 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010146 parsl.executors.status_handling DEBUG: Launched block 89 on executor multi with job ID 17010146 parsl.executors.status_handling INFO: Allocated block ID 90 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90 2024-03-13 09:46:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 2024-03-13 09:46:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-13 09:46:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef16fee211ffeb7276ef50cf93712bc7 with result from task 642 parsl.dataflow.dflow DEBUG: Task 642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 643 has memoization hash f8730a5b43d3b915447fb42ebcd60e25 parsl.dataflow.memoization INFO: Task 643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dd760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 643 try 0 launched on executor multi with executor id 554 parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/8e9ea480-804b-4c2d-a1f0-1ca1c6a37114_characterizeImage_898798_11.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/8e9ea480-804b-4c2d-a1f0-1ca1c6a37114_characterizeImage_898798_11.stderr parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/cb0a0edf-aa75-46ea-874e-4c6902bc94f2_isr_898798_11.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/cb0a0edf-aa75-46ea-874e-4c6902bc94f2_isr_898798_11.stderr parsl.dataflow.dflow INFO: Task 741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f8163c8c1e93c20af98b66a21c89537 with result from task 741 parsl.dataflow.dflow DEBUG: Task 741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 742 has memoization hash 201b534dd50008d065a3f9e127f1262b parsl.dataflow.memoization INFO: Task 742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c513100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 742 try 0 launched on executor multi with executor id 555 parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/0e90c368-ee12-4940-a195-be79d1145d14_characterizeImage_898694_24.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/0e90c368-ee12-4940-a195-be79d1145d14_characterizeImage_898694_24.stderr parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/e4491328-430f-4e87-bd88-e6e8f4fd58da_isr_898694_24.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/e4491328-430f-4e87-bd88-e6e8f4fd58da_isr_898694_24.stderr parsl.dataflow.dflow INFO: Task 693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65e465f975ffad16a15677f15812906a with result from task 693 parsl.dataflow.dflow DEBUG: Task 693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 694 has memoization hash a46556e6d33651853a9d4a93473d5b54 parsl.dataflow.memoization INFO: Task 694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3763e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 694 try 0 launched on executor multi with executor id 556 parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/7da9b342-a8e6-40bd-a79f-cd181a9bd133_characterizeImage_898694_23.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/7da9b342-a8e6-40bd-a79f-cd181a9bd133_characterizeImage_898694_23.stderr parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/be4bf9eb-4d4f-4e71-a458-475777ed1931_isr_898694_23.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/be4bf9eb-4d4f-4e71-a458-475777ed1931_isr_898694_23.stderr parsl.dataflow.dflow INFO: Task 585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e02c44ad56d7a32b6a6d33ee443107a0 with result from task 585 parsl.dataflow.dflow DEBUG: Task 585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 586 has memoization hash db73cd82ef2c749fab451944d18eebaf parsl.dataflow.memoization INFO: Task 586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4df100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 586 try 0 launched on executor multi with executor id 557 parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/17012c66-e273-4070-af33-c0c76f2fe7de_characterizeImage_898799_29.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/17012c66-e273-4070-af33-c0c76f2fe7de_characterizeImage_898799_29.stderr parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/bfb16a48-d7d4-4733-8ea6-118988daea34_isr_898799_29.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/bfb16a48-d7d4-4733-8ea6-118988daea34_isr_898799_29.stderr parsl.dataflow.dflow INFO: Task 702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 060a1b00385ac0dc29c33086f4d908a4 with result from task 702 parsl.dataflow.dflow DEBUG: Task 702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 703 has memoization hash cfdad89e8063d10156f94bfb7c872678 parsl.dataflow.memoization INFO: Task 703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c377240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 703 try 0 launched on executor multi with executor id 558 parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/364c5ceb-bb35-48aa-94ba-026f29b334b9_characterizeImage_898798_35.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/364c5ceb-bb35-48aa-94ba-026f29b334b9_characterizeImage_898798_35.stderr parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/ce856a54-5785-46a7-9c51-0dc0c6c9ef98_isr_898798_35.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/ce856a54-5785-46a7-9c51-0dc0c6c9ef98_isr_898798_35.stderr parsl.dataflow.dflow INFO: Task 705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0655ecfcf9da675165fa57d80f9f8d42 with result from task 705 parsl.dataflow.dflow DEBUG: Task 705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 706 has memoization hash 71f0cac51c0a7d73b9cc7325a7337bc4 parsl.dataflow.memoization INFO: Task 706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c377560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 706 try 0 launched on executor multi with executor id 559 parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/fd62df96-855b-4c72-82b0-8f1e161cb8e2_characterizeImage_898696_47.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/fd62df96-855b-4c72-82b0-8f1e161cb8e2_characterizeImage_898696_47.stderr parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/17215e85-5089-43da-8601-15ef276d3e2f_isr_898696_47.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/17215e85-5089-43da-8601-15ef276d3e2f_isr_898696_47.stderr parsl.dataflow.dflow INFO: Task 735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc7bd01c3159b4eb7607675277e1ab91 with result from task 735 parsl.dataflow.dflow DEBUG: Task 735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 736 has memoization hash e5a5e9fcd721475d2e071614a95870c0 parsl.dataflow.memoization INFO: Task 736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 736 try 0 launched on executor multi with executor id 560 parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/859c6268-5814-454b-b921-30f548078b3a_characterizeImage_898696_46.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/859c6268-5814-454b-b921-30f548078b3a_characterizeImage_898696_46.stderr parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/c2b9deea-e73d-44e0-a36b-b8b3d1875615_isr_898696_46.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/c2b9deea-e73d-44e0-a36b-b8b3d1875615_isr_898696_46.stderr parsl.dataflow.dflow INFO: Task 738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efa70df1e4b04f90673a6bf4d9768504 with result from task 738 parsl.dataflow.dflow DEBUG: Task 738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 739 has memoization hash 8cc23f314fa316fe15f411e407327730 parsl.dataflow.memoization INFO: Task 739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 739 try 0 launched on executor multi with executor id 561 parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/f172eb2e-c15b-45f6-a1ec-e8cabbc9015d_characterizeImage_898694_49.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/f172eb2e-c15b-45f6-a1ec-e8cabbc9015d_characterizeImage_898694_49.stderr parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/f1911837-c88d-4a24-9fd1-8bfe89fe6bed_isr_898694_49.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/f1911837-c88d-4a24-9fd1-8bfe89fe6bed_isr_898694_49.stderr parsl.dataflow.dflow INFO: Task 699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c51160c3aedf675ce026572b24552362 with result from task 699 parsl.dataflow.dflow DEBUG: Task 699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 700 has memoization hash 8eaaf86139ef20e9f5850c8bce020b0a parsl.dataflow.memoization INFO: Task 700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c375da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 700 try 0 launched on executor multi with executor id 562 parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/91f51146-7860-451d-8254-31e894198277_characterizeImage_898694_19.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/91f51146-7860-451d-8254-31e894198277_characterizeImage_898694_19.stderr parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/47692a71-04f2-47db-9589-5bb165d88617_isr_898694_19.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/47692a71-04f2-47db-9589-5bb165d88617_isr_898694_19.stderr parsl.dataflow.dflow INFO: Task 522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eef16923819fe152db234d974987f81f with result from task 522 parsl.dataflow.dflow DEBUG: Task 522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 523 has memoization hash 8057a1e552cebddd28e3a36ba08d47f5 parsl.dataflow.memoization INFO: Task 523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 563 parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/069409b1-6cf8-428a-9db6-be9f2eebf5ae_characterizeImage_898696_10.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/069409b1-6cf8-428a-9db6-be9f2eebf5ae_characterizeImage_898696_10.stderr parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/2d26d748-4826-4ac6-b2ac-778f62a73d2e_isr_898696_10.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/2d26d748-4826-4ac6-b2ac-778f62a73d2e_isr_898696_10.stderr parsl.dataflow.dflow INFO: Task 753 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39a6dc14cc93fd2292e9fae44de9cae0 with result from task 753 parsl.dataflow.dflow DEBUG: Task 753 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 754 has memoization hash c4a7fb5aea04dfbdbed2ead0171c0871 parsl.dataflow.memoization INFO: Task 754 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3ba660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 754 try 0 launched on executor multi with executor id 564 parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/7110b934-b1d4-4033-bd8c-020a49556d6d_characterizeImage_898694_20.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/7110b934-b1d4-4033-bd8c-020a49556d6d_characterizeImage_898694_20.stderr parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/fa07517d-faee-454e-aa09-365f78d8f2af_isr_898694_20.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/fa07517d-faee-454e-aa09-365f78d8f2af_isr_898694_20.stderr parsl.dataflow.dflow INFO: Task 744 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6aa1ef0fd74ede44ead26e8497dc08f2 with result from task 744 parsl.dataflow.dflow DEBUG: Task 744 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 745 has memoization hash d49705fd931b6259cf06ba5368575ec6 parsl.dataflow.memoization INFO: Task 745 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 745 try 0 launched on executor multi with executor id 565 parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/2c42c1e2-3d10-4c68-8e76-edd81a807591_characterizeImage_898798_47.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/2c42c1e2-3d10-4c68-8e76-edd81a807591_characterizeImage_898798_47.stderr parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/f75f1837-1b45-4abf-b78a-a4929e741f1e_isr_898798_47.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/f75f1837-1b45-4abf-b78a-a4929e741f1e_isr_898798_47.stderr parsl.dataflow.dflow INFO: Task 696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 659284bb359d590e56217cbcf121dadc with result from task 696 parsl.dataflow.dflow DEBUG: Task 696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 697 has memoization hash 0eb7d0c690bbb5b89c5606d34fdd2e5c parsl.dataflow.memoization INFO: Task 697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3760c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 697 try 0 launched on executor multi with executor id 566 parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/17efda62-0adf-4afe-8e6b-d2a6772f176f_characterizeImage_898800_52.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/17efda62-0adf-4afe-8e6b-d2a6772f176f_characterizeImage_898800_52.stderr parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/72fbb070-69fc-4fe1-823e-a7b3720505d8_isr_898800_52.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/72fbb070-69fc-4fe1-823e-a7b3720505d8_isr_898800_52.stderr parsl.dataflow.dflow INFO: Task 750 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d353042533a5b6e1e3f41611367fe41 with result from task 750 parsl.dataflow.dflow DEBUG: Task 750 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 751 has memoization hash ebdac79a2ca31b66f59c36b916c217ec parsl.dataflow.memoization INFO: Task 751 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3ba340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 751 try 0 launched on executor multi with executor id 567 parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/25b2c2d7-52e9-464b-a475-1e315326e128_characterizeImage_898800_43.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/25b2c2d7-52e9-464b-a475-1e315326e128_characterizeImage_898800_43.stderr parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/a47417d9-4462-4cc7-86da-8685e62efefb_isr_898800_43.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/a47417d9-4462-4cc7-86da-8685e62efefb_isr_898800_43.stderr 2024-03-13 09:46:41 proc_lsst.multi:170 [INFO] [multi] job_id 17010148 proc_lsst.multi INFO: [multi] job_id 17010148 2024-03-13 09:46:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-13 09:46:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010148 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010148 parsl.executors.status_handling DEBUG: Launched block 90 on executor multi with job ID 17010148 parsl.executors.status_handling INFO: Allocated block ID 91 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91 2024-03-13 09:46:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 2024-03-13 09:46:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-13 09:46:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 09:46:42 proc_lsst.multi:170 [INFO] [multi] job_id 17010149 proc_lsst.multi INFO: [multi] job_id 17010149 2024-03-13 09:46:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-13 09:46:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010149 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010149 parsl.executors.status_handling DEBUG: Launched block 91 on executor multi with job ID 17010149 parsl.executors.status_handling INFO: Allocated block ID 92 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92 2024-03-13 09:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 2024-03-13 09:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 09:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 09:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 09:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 09:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 09:46:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 09:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 09:46:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 09:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 09:46:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 09:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-13 09:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,198.48.92.26,169.254.95.120,10.64.129.9,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54743 --result_port=54001 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 747 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1747dedc13bf7382edf1a095f210a0b0 with result from task 747 parsl.dataflow.dflow DEBUG: Task 747 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 748 has memoization hash dfbb96fa1ba846f8ba31b9ea8b13087e parsl.dataflow.memoization INFO: Task 748 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 748 try 0 launched on executor multi with executor id 568 parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/5ea0781b-7114-4dc9-81a4-ebbb7c1db901_characterizeImage_898696_62.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/5ea0781b-7114-4dc9-81a4-ebbb7c1db901_characterizeImage_898696_62.stderr parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/e879ec44-0d3c-47fc-823f-71c504d93541_isr_898696_62.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/e879ec44-0d3c-47fc-823f-71c504d93541_isr_898696_62.stderr parsl.dataflow.dflow INFO: Task 756 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfcc785b1347b6d50039155b39b202a0 with result from task 756 parsl.dataflow.dflow DEBUG: Task 756 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 757 has memoization hash 143dc0abcdff39d7b5871741e898b92e parsl.dataflow.memoization INFO: Task 757 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3ba980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 757 try 0 launched on executor multi with executor id 569 parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/626adb04-9592-43bb-bef1-50faf374065d_characterizeImage_898800_39.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/626adb04-9592-43bb-bef1-50faf374065d_characterizeImage_898800_39.stderr parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/f91406e2-cdc0-40e7-91f4-9ca08a6e35f3_isr_898800_39.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/f91406e2-cdc0-40e7-91f4-9ca08a6e35f3_isr_898800_39.stderr 2024-03-13 09:46:43 proc_lsst.multi:170 [INFO] [multi] job_id 17010150 proc_lsst.multi INFO: [multi] job_id 17010150 2024-03-13 09:46:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 37 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37 2024-03-13 09:46:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 09:46:43 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17010150 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17010150 parsl.executors.status_handling DEBUG: Launched block 92 on executor multi with job ID 17010150 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 2024-03-13 09:46:43 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local parsl.dataflow.dflow INFO: Task 813 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd11bb7d230e539f6e42b968d092aa79 with result from task 813 parsl.dataflow.dflow DEBUG: Task 813 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 814 has memoization hash 2787d24ce3a552a407444111619de7fb parsl.dataflow.memoization INFO: Task 814 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 814 try 0 launched on executor multi with executor id 570 parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/de94a8c9-2e17-4e35-b05d-1858369f5633_characterizeImage_898799_21.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/de94a8c9-2e17-4e35-b05d-1858369f5633_characterizeImage_898799_21.stderr parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/2cc4d335-6b6c-4958-ab9e-559289881311_isr_898799_21.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/2cc4d335-6b6c-4958-ab9e-559289881311_isr_898799_21.stderr 2024-03-13 09:46:43 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49bd97e0e32012407abe24685fc298a4 with result from task 663 parsl.dataflow.dflow DEBUG: Task 663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 664 has memoization hash b39895c145d9c2f38f6a6458ac769dbb parsl.dataflow.memoization INFO: Task 664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c374040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 664 try 0 launched on executor multi with executor id 571 parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/0d8e4103-985c-4101-9969-cc0402388124_characterizeImage_898694_2.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/0d8e4103-985c-4101-9969-cc0402388124_characterizeImage_898694_2.stderr parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/3195a6fa-d59b-4c56-9815-ab6c46ad69e8_isr_898694_2.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/3195a6fa-d59b-4c56-9815-ab6c46ad69e8_isr_898694_2.stderr parsl.dataflow.dflow INFO: Task 759 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e2d95ff22cd18ca23694bd5e0842c9d with result from task 759 parsl.dataflow.dflow DEBUG: Task 759 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 760 has memoization hash eb1536b7ee76f2b48e6a1c46d480cee2 parsl.dataflow.memoization INFO: Task 760 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3baca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 760 try 0 launched on executor multi with executor id 572 parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/981007ea-b1bc-4d0e-90ec-94715bb9e83c_characterizeImage_898799_38.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/981007ea-b1bc-4d0e-90ec-94715bb9e83c_characterizeImage_898799_38.stderr parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/b62f2f07-2e20-4987-9de4-29314abd7104_isr_898799_38.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/b62f2f07-2e20-4987-9de4-29314abd7104_isr_898799_38.stderr parsl.dataflow.dflow INFO: Task 561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c48ad17fa68a4ac013d62c374fd2b16d with result from task 561 parsl.dataflow.dflow DEBUG: Task 561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 562 has memoization hash 8cc950de48dba5e9683430b2f3206cf0 parsl.dataflow.memoization INFO: Task 562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4ddd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 562 try 0 launched on executor multi with executor id 573 parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/7a3c6910-57fc-4a81-bbb3-85a8417d66fa_characterizeImage_898694_21.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/7a3c6910-57fc-4a81-bbb3-85a8417d66fa_characterizeImage_898694_21.stderr parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/4843ac7f-c248-4f2b-b658-79730fa7fa60_isr_898694_21.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/4843ac7f-c248-4f2b-b658-79730fa7fa60_isr_898694_21.stderr parsl.dataflow.dflow INFO: Task 897 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29df56e58bc6c40ec47039f749da6064 with result from task 897 parsl.dataflow.dflow DEBUG: Task 897 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 898 has memoization hash c2650775d3468113baf7b2a28af99e0c parsl.dataflow.memoization INFO: Task 898 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c443e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 898 try 0 launched on executor multi with executor id 574 parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/73b24cac-6b8e-44b5-b0e2-2eec52f96872_characterizeImage_898800_56.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/73b24cac-6b8e-44b5-b0e2-2eec52f96872_characterizeImage_898800_56.stderr parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/ee1d8302-0ca1-43d2-9d0c-7725a7328f9c_isr_898800_56.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/ee1d8302-0ca1-43d2-9d0c-7725a7328f9c_isr_898800_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:46:44 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry beb0b79893144b6888051408974cfac8 with result from task 582 parsl.dataflow.dflow DEBUG: Task 582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 583 has memoization hash 6e177f157f271a55972754aa73b11fa9 parsl.dataflow.memoization INFO: Task 583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 583 try 0 launched on executor multi with executor id 575 parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/4328e951-e6db-4b1c-976f-34b0cc418338_characterizeImage_898695_21.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/4328e951-e6db-4b1c-976f-34b0cc418338_characterizeImage_898695_21.stderr parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/1a3eacb6-b6d7-43dc-aa27-79c3f07a6114_isr_898695_21.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/1a3eacb6-b6d7-43dc-aa27-79c3f07a6114_isr_898695_21.stderr parsl.dataflow.dflow INFO: Task 660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25f52407c51321528cc226c2f6d6c71e with result from task 660 parsl.dataflow.dflow DEBUG: Task 660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 661 has memoization hash 5df32216f63aaa9d3ec01b0fe0b7cf79 parsl.dataflow.memoization INFO: Task 661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c513ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 661 try 0 launched on executor multi with executor id 576 parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/ff5d5a87-3752-446b-b651-b467c3e9ab77_characterizeImage_898694_16.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/ff5d5a87-3752-446b-b651-b467c3e9ab77_characterizeImage_898694_16.stderr parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/718752d7-cb50-41d5-a793-3e89258a2dd9_isr_898694_16.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/718752d7-cb50-41d5-a793-3e89258a2dd9_isr_898694_16.stderr parsl.dataflow.dflow INFO: Task 603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac1e6c35f11ea7b8ecac79bde03cd74d with result from task 603 parsl.dataflow.dflow DEBUG: Task 603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 604 has memoization hash e5f9493cbb9a6dc0aee9f32409c4b3d2 parsl.dataflow.memoization INFO: Task 604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c510ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 604 try 0 launched on executor multi with executor id 577 parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/297a28a5-09c7-43b1-b327-0d2bcfa45fa6_characterizeImage_898800_51.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/297a28a5-09c7-43b1-b327-0d2bcfa45fa6_characterizeImage_898800_51.stderr parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/d1b2154c-f053-4976-92c1-bf22f573a83b_isr_898800_51.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/d1b2154c-f053-4976-92c1-bf22f573a83b_isr_898800_51.stderr parsl.dataflow.dflow INFO: Task 573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e1009a0fa174baa05ed1378fd3bff57 with result from task 573 parsl.dataflow.dflow DEBUG: Task 573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 574 has memoization hash 741fd3767734dde779ed85e462b7da01 parsl.dataflow.memoization INFO: Task 574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4de520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 574 try 0 launched on executor multi with executor id 578 parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/fa3ca50b-e138-4ee0-81eb-606a4f103cb3_characterizeImage_898798_20.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/fa3ca50b-e138-4ee0-81eb-606a4f103cb3_characterizeImage_898798_20.stderr parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/9892d11f-0c1c-4c21-8a92-9f89b827b2bf_isr_898798_20.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/9892d11f-0c1c-4c21-8a92-9f89b827b2bf_isr_898798_20.stderr parsl.dataflow.dflow INFO: Task 567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 945dc8c4fbf97660854144d9763fba17 with result from task 567 parsl.dataflow.dflow DEBUG: Task 567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 568 has memoization hash 8d07b098be504e7b43198e67e70ffc27 parsl.dataflow.memoization INFO: Task 568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c5104a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 568 try 0 launched on executor multi with executor id 579 parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/5c8a5a13-7669-4145-8ebc-35340c188e17_characterizeImage_898800_49.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/5c8a5a13-7669-4145-8ebc-35340c188e17_characterizeImage_898800_49.stderr parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/9275adc8-1777-45fe-9e66-bb22aed5c3d4_isr_898800_49.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/9275adc8-1777-45fe-9e66-bb22aed5c3d4_isr_898800_49.stderr parsl.dataflow.dflow INFO: Task 624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55e59bf14eb4dd50aa9454e65fc8c027 with result from task 624 parsl.dataflow.dflow DEBUG: Task 624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 625 has memoization hash 55916d16015dfa032c752c5f6216e2ac parsl.dataflow.memoization INFO: Task 625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c5120c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 625 try 0 launched on executor multi with executor id 580 parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/23368ced-211e-49af-89a5-4f4878049a38_characterizeImage_898694_28.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/23368ced-211e-49af-89a5-4f4878049a38_characterizeImage_898694_28.stderr parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/2a294fc0-6d0e-43ca-a89e-d73e8825d5ec_isr_898694_28.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/2a294fc0-6d0e-43ca-a89e-d73e8825d5ec_isr_898694_28.stderr parsl.dataflow.dflow INFO: Task 597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abaafbdbf45c68f550cd0792ae5ee74f with result from task 597 parsl.dataflow.dflow DEBUG: Task 597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 598 has memoization hash 9f6a91a4c853fdd3f0c8d0a3da2c1350 parsl.dataflow.memoization INFO: Task 598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dfc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 598 try 0 launched on executor multi with executor id 581 parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/33b8bc78-2410-4bae-8ef8-2c1afaf6aa8a_characterizeImage_898799_41.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/33b8bc78-2410-4bae-8ef8-2c1afaf6aa8a_characterizeImage_898799_41.stderr parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/ff482b0f-2509-41fb-bc4d-0cb6b6baec01_isr_898799_41.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/ff482b0f-2509-41fb-bc4d-0cb6b6baec01_isr_898799_41.stderr parsl.dataflow.dflow INFO: Task 651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c4222227ad83d79970a056362ab4a27 with result from task 651 parsl.dataflow.dflow DEBUG: Task 651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 652 has memoization hash 257894ad9702ea584eb6a457551643cd parsl.dataflow.memoization INFO: Task 652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c513ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 652 try 0 launched on executor multi with executor id 582 parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/a6b856ac-9728-474b-a870-462982feb8d5_characterizeImage_898696_40.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/a6b856ac-9728-474b-a870-462982feb8d5_characterizeImage_898696_40.stderr parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/fdbc3fdd-6717-46ea-9a11-5c7e1f93b4ff_isr_898696_40.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/fdbc3fdd-6717-46ea-9a11-5c7e1f93b4ff_isr_898696_40.stderr parsl.dataflow.dflow INFO: Task 564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c9b73179dd38e116a44bc5de1759035 with result from task 564 parsl.dataflow.dflow DEBUG: Task 564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 565 has memoization hash b4299c550db0ff2d67e126a33e2a6232 parsl.dataflow.memoization INFO: Task 565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dfec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 565 try 0 launched on executor multi with executor id 583 parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/73021dd1-29da-4355-ac8d-213bdb1483e2_characterizeImage_898696_17.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/73021dd1-29da-4355-ac8d-213bdb1483e2_characterizeImage_898696_17.stderr parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/f1ee9a52-c80e-4971-ade2-a8589095180c_isr_898696_17.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/f1ee9a52-c80e-4971-ade2-a8589095180c_isr_898696_17.stderr parsl.dataflow.dflow INFO: Task 576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5936a8460c3e5221424cf6b47c9ef06b with result from task 576 parsl.dataflow.dflow DEBUG: Task 576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 577 has memoization hash f78fd1aacb43db0185e6b5935915973d parsl.dataflow.memoization INFO: Task 577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4de7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 577 try 0 launched on executor multi with executor id 584 parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/bc783b00-8aac-41a9-9814-18c4a7999657_characterizeImage_898695_16.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/bc783b00-8aac-41a9-9814-18c4a7999657_characterizeImage_898695_16.stderr parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/a1a655a6-7cde-4d2e-bfac-40e898026257_isr_898695_16.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/a1a655a6-7cde-4d2e-bfac-40e898026257_isr_898695_16.stderr parsl.dataflow.dflow INFO: Task 648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e75157e0a7030d3eeb3caad2ce304011 with result from task 648 parsl.dataflow.dflow DEBUG: Task 648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 649 has memoization hash 9184ce6e5ac2753d9ec0ad5e6c4f1217 parsl.dataflow.memoization INFO: Task 649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c513880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 649 try 0 launched on executor multi with executor id 585 parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/5b7518bf-ecca-445d-8280-79b6393113c1_characterizeImage_898798_18.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/5b7518bf-ecca-445d-8280-79b6393113c1_characterizeImage_898798_18.stderr parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/31bfde07-ee79-4f9a-9db6-2137cf2c5fa3_isr_898798_18.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/31bfde07-ee79-4f9a-9db6-2137cf2c5fa3_isr_898798_18.stderr parsl.dataflow.dflow INFO: Task 657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03873dcb559624ac487d89ffb01dc6b5 with result from task 657 parsl.dataflow.dflow DEBUG: Task 657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 658 has memoization hash 02135cf0533727b74fc36577ed1a6fec parsl.dataflow.memoization INFO: Task 658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c376a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 658 try 0 launched on executor multi with executor id 586 parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/e6ae0ab7-345c-40ca-ac11-9d65cec3c4a7_characterizeImage_898696_22.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/e6ae0ab7-345c-40ca-ac11-9d65cec3c4a7_characterizeImage_898696_22.stderr parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/626dba7b-dfa5-4fa1-849c-d3e968e04f7c_isr_898696_22.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/626dba7b-dfa5-4fa1-849c-d3e968e04f7c_isr_898696_22.stderr parsl.dataflow.dflow INFO: Task 594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc9ed16f0c312816559977e48b2c63d8 with result from task 594 parsl.dataflow.dflow DEBUG: Task 594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 595 has memoization hash aa6d8b98fb270d176210c2b4f8e9aea3 parsl.dataflow.memoization INFO: Task 595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dfba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 595 try 0 launched on executor multi with executor id 587 parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/6dcb40de-2331-4005-b84f-5183df3feafb_characterizeImage_898694_45.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/6dcb40de-2331-4005-b84f-5183df3feafb_characterizeImage_898694_45.stderr parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/8411a991-616f-4d1a-a90c-d58598becb1a_isr_898694_45.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/8411a991-616f-4d1a-a90c-d58598becb1a_isr_898694_45.stderr parsl.dataflow.dflow INFO: Task 591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b81eafbbd195c45b77d3fb5f2fc825e9 with result from task 591 parsl.dataflow.dflow DEBUG: Task 591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 592 has memoization hash 0c407ea5eeb38e87070cec824193467e parsl.dataflow.memoization INFO: Task 592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4df880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 592 try 0 launched on executor multi with executor id 588 parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/c2c98fcb-ce66-4a4f-9d4a-c3db3a7d9ff3_characterizeImage_898798_25.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/c2c98fcb-ce66-4a4f-9d4a-c3db3a7d9ff3_characterizeImage_898798_25.stderr parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/aea39453-e90a-4d09-9dd6-65a8e2b65b98_isr_898798_25.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/aea39453-e90a-4d09-9dd6-65a8e2b65b98_isr_898798_25.stderr parsl.dataflow.dflow INFO: Task 627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a8b20ee32725ad0c3debda2a0a3cdd0 with result from task 627 parsl.dataflow.dflow DEBUG: Task 627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 628 has memoization hash c58dd320d85787eb54949d1eba2bc316 parsl.dataflow.memoization INFO: Task 628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c5123e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 628 try 0 launched on executor multi with executor id 589 parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/3785309e-d272-4d88-bb97-415daec530d3_characterizeImage_898799_34.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/3785309e-d272-4d88-bb97-415daec530d3_characterizeImage_898799_34.stderr parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/137efc9f-66b3-4d9d-b06a-3303abf4e75b_isr_898799_34.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/137efc9f-66b3-4d9d-b06a-3303abf4e75b_isr_898799_34.stderr parsl.dataflow.dflow INFO: Task 570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efaa72423265df7a63f269f9a5211989 with result from task 570 parsl.dataflow.dflow DEBUG: Task 570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 571 has memoization hash cb26be1d1d7ff6290f152c7996aeceeb parsl.dataflow.memoization INFO: Task 571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4ddee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 571 try 0 launched on executor multi with executor id 590 parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/ba463b6a-e421-49d4-96f4-b4fec073c5ac_characterizeImage_898799_26.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/ba463b6a-e421-49d4-96f4-b4fec073c5ac_characterizeImage_898799_26.stderr parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/b7d9e3d5-6672-4a15-bb64-4b3f8926db14_isr_898799_26.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/b7d9e3d5-6672-4a15-bb64-4b3f8926db14_isr_898799_26.stderr parsl.dataflow.dflow INFO: Task 633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5241a87baa9e582ff9aef920e6b1b9fa with result from task 633 parsl.dataflow.dflow DEBUG: Task 633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 634 has memoization hash 45d18d5f63d33d03a06c7119cc0b432e parsl.dataflow.memoization INFO: Task 634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c512a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 634 try 0 launched on executor multi with executor id 591 parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/8482838e-3f08-48d6-a578-ee12e41a83cb_characterizeImage_898798_45.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/8482838e-3f08-48d6-a578-ee12e41a83cb_characterizeImage_898798_45.stderr parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/593418a7-f6e0-46c3-a570-d0090b216e2e_isr_898798_45.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/593418a7-f6e0-46c3-a570-d0090b216e2e_isr_898798_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:46:46 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a14486862bc70b911b49aab3b2197d79 with result from task 639 parsl.dataflow.dflow DEBUG: Task 639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 640 has memoization hash 5defc5391d5b5dac2bd71eb2394a80a7 parsl.dataflow.memoization INFO: Task 640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c513060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 640 try 0 launched on executor multi with executor id 592 parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/5dd416c4-9f42-4d0d-92d7-b499c434cf66_characterizeImage_898799_62.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/5dd416c4-9f42-4d0d-92d7-b499c434cf66_characterizeImage_898799_62.stderr parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/1d1b85a9-ecd9-4b60-8632-9bdd72cd7961_isr_898799_62.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/1d1b85a9-ecd9-4b60-8632-9bdd72cd7961_isr_898799_62.stderr parsl.dataflow.dflow INFO: Task 654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffbf45ac7a69c75c3410595002075e92 with result from task 654 parsl.dataflow.dflow DEBUG: Task 654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 655 has memoization hash ce7d2bf51d96310945a65b54d94fa4d3 parsl.dataflow.memoization INFO: Task 655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c376f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 655 try 0 launched on executor multi with executor id 593 parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/b7f1116a-03a2-4392-b6fa-e343aedfbd10_characterizeImage_898800_45.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/b7f1116a-03a2-4392-b6fa-e343aedfbd10_characterizeImage_898800_45.stderr parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/d5a1807d-cc06-4855-9b1c-8109e1ce01ba_isr_898800_45.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/d5a1807d-cc06-4855-9b1c-8109e1ce01ba_isr_898800_45.stderr parsl.dataflow.dflow INFO: Task 606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0e83e0f6ca540b91fa698fe2c7fc716 with result from task 606 parsl.dataflow.dflow DEBUG: Task 606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 607 has memoization hash 376be6dc4da82ff80e52f553bde9a7f4 parsl.dataflow.memoization INFO: Task 607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c510e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 607 try 0 launched on executor multi with executor id 594 parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/ea912538-89ff-4622-87e2-ba0e5ae3f1aa_characterizeImage_898696_27.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/ea912538-89ff-4622-87e2-ba0e5ae3f1aa_characterizeImage_898696_27.stderr parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/6b725fec-a4d3-43cd-a74f-ec666a7e3da1_isr_898696_27.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/6b725fec-a4d3-43cd-a74f-ec666a7e3da1_isr_898696_27.stderr parsl.dataflow.dflow INFO: Task 666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7e387ebc4ac0049932aed1e131f400c with result from task 666 parsl.dataflow.dflow DEBUG: Task 666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 667 has memoization hash 3975a35ad57dae249c42f475313db75a parsl.dataflow.memoization INFO: Task 667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c374400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 667 try 0 launched on executor multi with executor id 595 parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/6a282091-7d72-4dd6-b8c0-3517d70f0f1e_characterizeImage_898800_11.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/6a282091-7d72-4dd6-b8c0-3517d70f0f1e_characterizeImage_898800_11.stderr parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/a92c0fa2-6efc-4148-93f4-e8e0e174ca3d_isr_898800_11.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/a92c0fa2-6efc-4148-93f4-e8e0e174ca3d_isr_898800_11.stderr parsl.dataflow.dflow INFO: Task 681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6967e2d3bbcf7b03cf0ed4f5a5bae5c5 with result from task 681 parsl.dataflow.dflow DEBUG: Task 681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 682 has memoization hash 7c2d9345a6cae992e283a125fc0c3108 parsl.dataflow.memoization INFO: Task 682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c375260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 682 try 0 launched on executor multi with executor id 596 parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/72d3e4dd-f3ea-4611-9daf-34e9c71ce583_characterizeImage_898799_31.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/72d3e4dd-f3ea-4611-9daf-34e9c71ce583_characterizeImage_898799_31.stderr parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/b559c7e2-f1eb-41ef-996d-3f7783bd3522_isr_898799_31.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/b559c7e2-f1eb-41ef-996d-3f7783bd3522_isr_898799_31.stderr parsl.dataflow.dflow INFO: Task 819 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2397d361a6060eb24bfe566829b183e8 with result from task 819 parsl.dataflow.dflow DEBUG: Task 819 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 820 has memoization hash 1a49633e619abcde7421b88f067b3b49 parsl.dataflow.memoization INFO: Task 820 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 820 try 0 launched on executor multi with executor id 597 parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/2f0434f7-8a50-4bfb-922d-11e4088ed3a5_characterizeImage_898696_51.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/2f0434f7-8a50-4bfb-922d-11e4088ed3a5_characterizeImage_898696_51.stderr parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/7c794282-6c86-4217-874b-99acab999514_isr_898696_51.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/7c794282-6c86-4217-874b-99acab999514_isr_898696_51.stderr parsl.dataflow.dflow INFO: Task 900 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9141156de10cd5160a8069ba29800e0f with result from task 900 parsl.dataflow.dflow DEBUG: Task 900 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 901 has memoization hash 2e2097327dfec86645c84278b85823be parsl.dataflow.memoization INFO: Task 901 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3bad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 901 try 0 launched on executor multi with executor id 598 parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/5d91dbe4-b071-4fee-aeaa-671ec9123def_characterizeImage_898800_34.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/5d91dbe4-b071-4fee-aeaa-671ec9123def_characterizeImage_898800_34.stderr parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/94ec93e4-5de8-4071-bcbe-d96c0c5c52c3_isr_898800_34.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/94ec93e4-5de8-4071-bcbe-d96c0c5c52c3_isr_898800_34.stderr parsl.dataflow.dflow INFO: Task 636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9042284bde263c62331a6f8969df1e64 with result from task 636 parsl.dataflow.dflow DEBUG: Task 636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 637 has memoization hash f6565e32a853e0577e8ffe73abc395ab parsl.dataflow.memoization INFO: Task 637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c512d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 637 try 0 launched on executor multi with executor id 599 parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/c1adc23c-864b-4eb7-96d3-a034c1d7018c_characterizeImage_898799_13.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/c1adc23c-864b-4eb7-96d3-a034c1d7018c_characterizeImage_898799_13.stderr parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/37aa4699-3c96-40b8-a467-c42a8d857693_isr_898799_13.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/37aa4699-3c96-40b8-a467-c42a8d857693_isr_898799_13.stderr parsl.dataflow.dflow INFO: Task 579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e982bc6f6135509923514c0d52317f4 with result from task 579 parsl.dataflow.dflow DEBUG: Task 579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 580 has memoization hash e6edfbaa8a43a6cc5b68e24ce80ec1bd parsl.dataflow.memoization INFO: Task 580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 580 try 0 launched on executor multi with executor id 600 parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/684e423b-7138-4544-92c6-f77f1e7817d1_characterizeImage_898798_60.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/684e423b-7138-4544-92c6-f77f1e7817d1_characterizeImage_898798_60.stderr parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/070f1e53-fad0-4516-b5d3-38702c819341_isr_898798_60.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/070f1e53-fad0-4516-b5d3-38702c819341_isr_898798_60.stderr parsl.dataflow.dflow INFO: Task 630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9ad7e935b8267d00ac08144e66fab46 with result from task 630 parsl.dataflow.dflow DEBUG: Task 630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 631 has memoization hash 431ee53e06ce383c7a6058b68c897eaf parsl.dataflow.memoization INFO: Task 631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c512700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 631 try 0 launched on executor multi with executor id 601 parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/b13f33ae-58f9-4457-b9c0-ab37d39c4266_characterizeImage_898696_49.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/b13f33ae-58f9-4457-b9c0-ab37d39c4266_characterizeImage_898696_49.stderr parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/940eb93e-802d-4282-9afd-95e8a1ce481c_isr_898696_49.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/940eb93e-802d-4282-9afd-95e8a1ce481c_isr_898696_49.stderr parsl.dataflow.dflow INFO: Task 588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09f8faa7167bb5ec65bf72cfb333d236 with result from task 588 parsl.dataflow.dflow DEBUG: Task 588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 589 has memoization hash 727dc94e0a4e098a243f8648c1a47278 parsl.dataflow.memoization INFO: Task 589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4df420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 589 try 0 launched on executor multi with executor id 602 parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/679acd56-4331-47fd-a2a1-2b9bb69f70a3_characterizeImage_898695_24.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/679acd56-4331-47fd-a2a1-2b9bb69f70a3_characterizeImage_898695_24.stderr parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/4eb8029a-1caa-4d93-85dc-ca6cd1144f58_isr_898695_24.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/4eb8029a-1caa-4d93-85dc-ca6cd1144f58_isr_898695_24.stderr parsl.dataflow.dflow INFO: Task 894 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry baaa506d45dec6e18793892b4a64196a with result from task 894 parsl.dataflow.dflow DEBUG: Task 894 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 895 has memoization hash aa97e55f61023b971683fcab46ce195f parsl.dataflow.memoization INFO: Task 895 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c443b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 895 try 0 launched on executor multi with executor id 603 parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/895d094b-d305-488b-8403-17510bdd1469_characterizeImage_898798_34.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/895d094b-d305-488b-8403-17510bdd1469_characterizeImage_898798_34.stderr parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/078acad5-b275-4475-a1e6-a88236b122b8_isr_898798_34.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/078acad5-b275-4475-a1e6-a88236b122b8_isr_898798_34.stderr parsl.dataflow.dflow INFO: Task 669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ddeef2b9d2acf2f4796ec26486257c8 with result from task 669 parsl.dataflow.dflow DEBUG: Task 669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 670 has memoization hash 5b98794a6d2eb1f1e9b63e967f2960c1 parsl.dataflow.memoization INFO: Task 670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c374720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 670 try 0 launched on executor multi with executor id 604 parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/0774fad2-e5d5-493c-9daf-1337a6f7e800_characterizeImage_898799_1.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/0774fad2-e5d5-493c-9daf-1337a6f7e800_characterizeImage_898799_1.stderr parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/435713ce-838b-442b-8f63-90567e447b56_isr_898799_1.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/435713ce-838b-442b-8f63-90567e447b56_isr_898799_1.stderr parsl.dataflow.dflow INFO: Task 1008 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 599d1a91e2458b4e96cf0345e512b3ad with result from task 1008 parsl.dataflow.dflow DEBUG: Task 1008 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1009 has memoization hash 7cdec4012c9852d23b211abb235d1efc parsl.dataflow.memoization INFO: Task 1009 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2db240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1009 try 0 launched on executor multi with executor id 605 parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/3aab0bda-1981-4fb6-ae9e-397916d8dd80_characterizeImage_898798_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/3aab0bda-1981-4fb6-ae9e-397916d8dd80_characterizeImage_898798_8.stderr parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/8f81b55f-fc28-47a4-bbe7-38d5d32781e9_isr_898798_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/8f81b55f-fc28-47a4-bbe7-38d5d32781e9_isr_898798_8.stderr parsl.dataflow.dflow INFO: Task 600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3835a5165ef4674b5a979277a5385892 with result from task 600 parsl.dataflow.dflow DEBUG: Task 600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 601 has memoization hash a20750f439edb585258e97d1175502f1 parsl.dataflow.memoization INFO: Task 601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c510720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 601 try 0 launched on executor multi with executor id 606 parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/53ab60d2-2cf3-4c46-9374-602242198b08_characterizeImage_898800_23.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/53ab60d2-2cf3-4c46-9374-602242198b08_characterizeImage_898800_23.stderr parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/0fa6f325-28bf-4d7f-8fbe-71931bef0822_isr_898800_23.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/0fa6f325-28bf-4d7f-8fbe-71931bef0822_isr_898800_23.stderr parsl.dataflow.dflow INFO: Task 690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05c4b090d2a2bc5cd6145a47b74ac268 with result from task 690 parsl.dataflow.dflow DEBUG: Task 690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 691 has memoization hash 50167c6be7de79bb172c7f303d03abc5 parsl.dataflow.memoization INFO: Task 691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3767a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 691 try 0 launched on executor multi with executor id 607 parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/9473746a-381a-42a6-a0a6-d63594874c2b_characterizeImage_898695_34.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/9473746a-381a-42a6-a0a6-d63594874c2b_characterizeImage_898695_34.stderr parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/3821b160-ee48-44aa-af6c-d985438f7781_isr_898695_34.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/3821b160-ee48-44aa-af6c-d985438f7781_isr_898695_34.stderr parsl.dataflow.dflow INFO: Task 675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c28a1c81faf47cd51524834a3c99d0af with result from task 675 parsl.dataflow.dflow DEBUG: Task 675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 676 has memoization hash 6b00d60a8b2d4de58447168ea5c322b7 parsl.dataflow.memoization INFO: Task 676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c374c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 676 try 0 launched on executor multi with executor id 608 parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/3b9cf0b0-6304-4fcd-a085-ca8f809d8957_characterizeImage_898695_1.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/3b9cf0b0-6304-4fcd-a085-ca8f809d8957_characterizeImage_898695_1.stderr parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/5b0d054f-72bc-4559-8850-d3278bf4d9e2_isr_898695_1.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/5b0d054f-72bc-4559-8850-d3278bf4d9e2_isr_898695_1.stderr parsl.dataflow.dflow INFO: Task 882 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e990a55c07a0599231c92d22305a91c with result from task 882 parsl.dataflow.dflow DEBUG: Task 882 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 883 has memoization hash 89322ae988c8e38d298f4f6f11f01cd7 parsl.dataflow.memoization INFO: Task 883 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c442e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 883 try 0 launched on executor multi with executor id 609 parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/e157aee2-b62e-4d1a-a5fb-6c0c83755762_characterizeImage_898798_22.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/e157aee2-b62e-4d1a-a5fb-6c0c83755762_characterizeImage_898798_22.stderr parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/6f265980-2235-4801-ab94-c436bf47de23_isr_898798_22.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/6f265980-2235-4801-ab94-c436bf47de23_isr_898798_22.stderr parsl.dataflow.dflow INFO: Task 549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9125eaa8789fce4778d61eeb042f288 with result from task 549 parsl.dataflow.dflow DEBUG: Task 549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 550 has memoization hash 2f0099ea4150e3abd38cb3d6bd6953de parsl.dataflow.memoization INFO: Task 550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dd080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 610 parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/46f6f883-8578-4567-8689-8b699ed76217_characterizeImage_898799_28.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/46f6f883-8578-4567-8689-8b699ed76217_characterizeImage_898799_28.stderr parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/59aa80e5-0751-4754-9395-2094a52c78fe_isr_898799_28.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/59aa80e5-0751-4754-9395-2094a52c78fe_isr_898799_28.stderr parsl.dataflow.dflow INFO: Task 825 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d18791b0eaf55bb7451bfeca5072d4f5 with result from task 825 parsl.dataflow.dflow DEBUG: Task 825 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 826 has memoization hash 93f8fc1681c49e3d512a8deb210def6e parsl.dataflow.memoization INFO: Task 826 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 826 try 0 launched on executor multi with executor id 611 parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/9366a3d6-bcbb-43e6-8183-e796da08819d_characterizeImage_898696_39.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/9366a3d6-bcbb-43e6-8183-e796da08819d_characterizeImage_898696_39.stderr parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/645d3164-4768-43a3-9c07-10a42976d8c1_isr_898696_39.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/645d3164-4768-43a3-9c07-10a42976d8c1_isr_898696_39.stderr parsl.dataflow.dflow INFO: Task 828 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 441c2100b9d566354feaadf06d0329ac with result from task 828 parsl.dataflow.dflow DEBUG: Task 828 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 829 has memoization hash ac8acbe0ced07e07d16c0bc44f42eada parsl.dataflow.memoization INFO: Task 829 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 829 try 0 launched on executor multi with executor id 612 parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/bb7619d4-3a63-4078-82f9-3bfe7d260ebc_characterizeImage_898694_34.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/bb7619d4-3a63-4078-82f9-3bfe7d260ebc_characterizeImage_898694_34.stderr parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/c85ec6d2-df10-4915-8ed0-f3ff55499d1e_isr_898694_34.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/c85ec6d2-df10-4915-8ed0-f3ff55499d1e_isr_898694_34.stderr parsl.dataflow.dflow INFO: Task 885 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c644e2ac49b20256f9468bd38272a69a with result from task 885 parsl.dataflow.dflow DEBUG: Task 885 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 886 has memoization hash a9c9530ce9cc4c1479faf80adc44cb01 parsl.dataflow.memoization INFO: Task 886 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4431a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 886 try 0 launched on executor multi with executor id 613 parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/63ce5329-c736-467a-92c4-e84a9730e459_characterizeImage_898799_14.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/63ce5329-c736-467a-92c4-e84a9730e459_characterizeImage_898799_14.stderr parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/ae1db5e7-3eeb-4b3e-b50d-492fc6c37434_isr_898799_14.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/ae1db5e7-3eeb-4b3e-b50d-492fc6c37434_isr_898799_14.stderr parsl.dataflow.dflow INFO: Task 876 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d6a6a0ea0320652a32a9a2fe7e1f68d with result from task 876 parsl.dataflow.dflow DEBUG: Task 876 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 877 has memoization hash 69f94db51b6a4eeabb666fc407ca08d2 parsl.dataflow.memoization INFO: Task 877 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c442840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 877 try 0 launched on executor multi with executor id 614 parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/f73fcf90-b226-4427-83ef-0ee48a96341c_characterizeImage_898696_25.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/f73fcf90-b226-4427-83ef-0ee48a96341c_characterizeImage_898696_25.stderr parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/4b3933b2-7b06-417b-85ae-718c8ed1c25b_isr_898696_25.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/4b3933b2-7b06-417b-85ae-718c8ed1c25b_isr_898696_25.stderr parsl.dataflow.dflow INFO: Task 843 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 337b311920e2566c9ad6cfbf1ebf3dff with result from task 843 parsl.dataflow.dflow DEBUG: Task 843 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 844 has memoization hash 887f7d8f1270253f3b462ca28ddb911e parsl.dataflow.memoization INFO: Task 844 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4405e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 844 try 0 launched on executor multi with executor id 615 parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/aca03799-8d72-4d70-b878-d6ee08834078_characterizeImage_898798_42.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/aca03799-8d72-4d70-b878-d6ee08834078_characterizeImage_898798_42.stderr parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/14a96df3-3d35-47e3-b334-2461603716ed_isr_898798_42.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/14a96df3-3d35-47e3-b334-2461603716ed_isr_898798_42.stderr parsl.dataflow.dflow INFO: Task 621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fe5141fe808dee2f7ecaeb087eeca65 with result from task 621 parsl.dataflow.dflow DEBUG: Task 621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 622 has memoization hash 662e1e68b27bdf0d1de2d67a20cc45f5 parsl.dataflow.memoization INFO: Task 622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c511da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 622 try 0 launched on executor multi with executor id 616 parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/19ffa1ec-f41d-4f29-86d2-8381cb651c8c_characterizeImage_898696_4.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/19ffa1ec-f41d-4f29-86d2-8381cb651c8c_characterizeImage_898696_4.stderr parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/fa6d8d5c-54f9-4eac-989c-8d604a1f5ea3_isr_898696_4.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/fa6d8d5c-54f9-4eac-989c-8d604a1f5ea3_isr_898696_4.stderr parsl.dataflow.dflow INFO: Task 879 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d444490d4ec3699f92444658c851a6ce with result from task 879 parsl.dataflow.dflow DEBUG: Task 879 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 880 has memoization hash 5967196e86dbcd02374105454419dbdc parsl.dataflow.memoization INFO: Task 880 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c442b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 880 try 0 launched on executor multi with executor id 617 parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ee8cb9d6-20ff-4625-a41b-e85548f00bbf_characterizeImage_898798_3.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ee8cb9d6-20ff-4625-a41b-e85548f00bbf_characterizeImage_898798_3.stderr parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/b66bf1f5-5ec7-4775-9851-a3e10a4ba270_isr_898798_3.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/b66bf1f5-5ec7-4775-9851-a3e10a4ba270_isr_898798_3.stderr parsl.dataflow.dflow INFO: Task 762 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1abd421d8fc58c88160d52b6c7131001 with result from task 762 parsl.dataflow.dflow DEBUG: Task 762 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 763 has memoization hash 69d724c666e80fd3a500aa9f875e8ac6 parsl.dataflow.memoization INFO: Task 763 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3baf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:46:47 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Parsl task 763 try 0 launched on executor multi with executor id 618 parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/635a5f6a-c7c9-465f-bfbf-22e5d2f43add_characterizeImage_898798_49.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/635a5f6a-c7c9-465f-bfbf-22e5d2f43add_characterizeImage_898798_49.stderr parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/acd945bd-661a-4276-8f96-a4fa1e36e993_isr_898798_49.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/acd945bd-661a-4276-8f96-a4fa1e36e993_isr_898798_49.stderr parsl.dataflow.dflow INFO: Task 849 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 094e566f6134a93f4977590a12fbd236 with result from task 849 parsl.dataflow.dflow DEBUG: Task 849 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 850 has memoization hash a75c23c1d66ab4d9ed827e0ebf204ab1 parsl.dataflow.memoization INFO: Task 850 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c440c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 850 try 0 launched on executor multi with executor id 619 parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/b9945013-2e60-4ee1-951f-164928e8e983_characterizeImage_898799_56.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/b9945013-2e60-4ee1-951f-164928e8e983_characterizeImage_898799_56.stderr parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/5d061782-5758-47a5-9150-c00756cd10dc_isr_898799_56.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/5d061782-5758-47a5-9150-c00756cd10dc_isr_898799_56.stderr parsl.dataflow.dflow INFO: Task 645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8a1b3a951df637a24f70dd60332e4ef with result from task 645 parsl.dataflow.dflow DEBUG: Task 645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 646 has memoization hash 92435ff8ae64041088cfe8442f47f402 parsl.dataflow.memoization INFO: Task 646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c5134c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 646 try 0 launched on executor multi with executor id 620 parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/3d2ef5e1-90f4-4971-94e2-b3127268545a_characterizeImage_898799_3.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/3d2ef5e1-90f4-4971-94e2-b3127268545a_characterizeImage_898799_3.stderr parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/65701232-472a-42d1-939d-200a44e08ddd_isr_898799_3.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/65701232-472a-42d1-939d-200a44e08ddd_isr_898799_3.stderr parsl.dataflow.dflow INFO: Task 840 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7fd676fa0659213a050229c6f92b700 with result from task 840 parsl.dataflow.dflow DEBUG: Task 840 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 841 has memoization hash 677a066dbf70efb0c278b0e9a8e48509 parsl.dataflow.memoization INFO: Task 841 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4402c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 841 try 0 launched on executor multi with executor id 621 parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/6fa874dd-14e1-4366-a53a-b2f4652e25ab_characterizeImage_898696_14.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/6fa874dd-14e1-4366-a53a-b2f4652e25ab_characterizeImage_898696_14.stderr parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/7b2152f8-3942-4a6b-a237-324d28b51a91_isr_898696_14.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/7b2152f8-3942-4a6b-a237-324d28b51a91_isr_898696_14.stderr parsl.dataflow.dflow INFO: Task 822 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96b8812222f8c85914cb428929cd09a2 with result from task 822 parsl.dataflow.dflow DEBUG: Task 822 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 823 has memoization hash b29697de76ebe2bfa759556bfff11c03 parsl.dataflow.memoization INFO: Task 823 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 823 try 0 launched on executor multi with executor id 622 parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/0b491b7d-f470-47da-b801-1add5d03aaae_characterizeImage_898800_17.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/0b491b7d-f470-47da-b801-1add5d03aaae_characterizeImage_898800_17.stderr parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/2280e2d9-565c-4f49-8e8d-f7c781bbaba1_isr_898800_17.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/2280e2d9-565c-4f49-8e8d-f7c781bbaba1_isr_898800_17.stderr parsl.dataflow.dflow INFO: Task 543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b69360722706345a49ef9c367b887613 with result from task 543 parsl.dataflow.dflow DEBUG: Task 543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 544 has memoization hash fea4ca5152b8eac8a5866c2f4332ecdd parsl.dataflow.memoization INFO: Task 544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4819e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 623 parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/71a9e1af-f47b-4d79-b7e2-23e2eef24fdd_characterizeImage_898799_48.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/71a9e1af-f47b-4d79-b7e2-23e2eef24fdd_characterizeImage_898799_48.stderr parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/1b219a5e-8018-4444-9943-992a91c4a17e_isr_898799_48.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/1b219a5e-8018-4444-9943-992a91c4a17e_isr_898799_48.stderr parsl.dataflow.dflow INFO: Task 672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c50fb76df17ee4dd471641b15dc9331d with result from task 672 parsl.dataflow.dflow DEBUG: Task 672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 673 has memoization hash 980247401cc597902c48e5a6591cdaf6 parsl.dataflow.memoization INFO: Task 673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c374a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 673 try 0 launched on executor multi with executor id 624 parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/8dfdd9dc-5aa7-43b7-8ba0-90a0a6f262ff_characterizeImage_898694_43.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/8dfdd9dc-5aa7-43b7-8ba0-90a0a6f262ff_characterizeImage_898694_43.stderr parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/d9efea15-1c5e-449b-a9a1-266822831fb2_isr_898694_43.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/d9efea15-1c5e-449b-a9a1-266822831fb2_isr_898694_43.stderr parsl.dataflow.dflow INFO: Task 678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fbc38488043bb33d8659288269f5fb3 with result from task 678 parsl.dataflow.dflow DEBUG: Task 678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 679 has memoization hash 80e19141a38bbc3c347048a5d599d1fc parsl.dataflow.memoization INFO: Task 679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c374f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 679 try 0 launched on executor multi with executor id 625 parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/a1ecec3d-6d4b-41da-8896-db74a4c30379_characterizeImage_898696_44.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/a1ecec3d-6d4b-41da-8896-db74a4c30379_characterizeImage_898696_44.stderr parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/19aedd40-ecf7-4fe4-a36c-975cf41e959b_isr_898696_44.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/19aedd40-ecf7-4fe4-a36c-975cf41e959b_isr_898696_44.stderr parsl.dataflow.dflow INFO: Task 618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df1a068506828b9d56e3e17421aa3182 with result from task 618 parsl.dataflow.dflow DEBUG: Task 618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 619 has memoization hash c861ce7e8809b9d786381b7531ea838d parsl.dataflow.memoization INFO: Task 619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c511a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 619 try 0 launched on executor multi with executor id 626 parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/e1dfe223-522c-410f-895f-4e44e59b736a_characterizeImage_898800_38.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/e1dfe223-522c-410f-895f-4e44e59b736a_characterizeImage_898800_38.stderr parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/0b79bf7d-68dc-4c56-83eb-9c764020e18d_isr_898800_38.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/0b79bf7d-68dc-4c56-83eb-9c764020e18d_isr_898800_38.stderr parsl.dataflow.dflow INFO: Task 834 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72d956b0374cfdb4af2a146281a192b1 with result from task 834 parsl.dataflow.dflow DEBUG: Task 834 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 835 has memoization hash 9f389e73fa0cf0a5fbebd01e51dfe495 parsl.dataflow.memoization INFO: Task 835 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 835 try 0 launched on executor multi with executor id 627 parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/23c8dcc2-7993-4e10-9c73-f65123cca940_characterizeImage_898799_40.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/23c8dcc2-7993-4e10-9c73-f65123cca940_characterizeImage_898799_40.stderr parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/d425a1e1-3548-43f7-8ad9-9c5d104f3618_isr_898799_40.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/d425a1e1-3548-43f7-8ad9-9c5d104f3618_isr_898799_40.stderr parsl.dataflow.dflow INFO: Task 684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66acb793e2ffd287f5abe6da706ceef2 with result from task 684 parsl.dataflow.dflow DEBUG: Task 684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 685 has memoization hash 52b97924429232f45030d5ee95a3ef91 parsl.dataflow.memoization INFO: Task 685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3756c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 685 try 0 launched on executor multi with executor id 628 parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/7b3cd1d6-c316-4af7-bd9c-0494d222cc41_characterizeImage_898799_50.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/7b3cd1d6-c316-4af7-bd9c-0494d222cc41_characterizeImage_898799_50.stderr parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/2183e6d4-2365-4e55-b7e3-00bdb4a7ca1a_isr_898799_50.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/2183e6d4-2365-4e55-b7e3-00bdb4a7ca1a_isr_898799_50.stderr parsl.dataflow.dflow INFO: Task 1002 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7374066c9cc9d3b9a2081043001493c2 with result from task 1002 parsl.dataflow.dflow DEBUG: Task 1002 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1003 has memoization hash 3a6c16b5c810ba0e2a37790a972d5d82 parsl.dataflow.memoization INFO: Task 1003 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2dac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1003 try 0 launched on executor multi with executor id 629 parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/bf09297e-f6f0-474a-921f-aecf3b613097_characterizeImage_898799_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/bf09297e-f6f0-474a-921f-aecf3b613097_characterizeImage_898799_53.stderr parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/570da354-59d2-4e18-a705-9d622317b64c_isr_898799_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/570da354-59d2-4e18-a705-9d622317b64c_isr_898799_53.stderr parsl.dataflow.dflow INFO: Task 816 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6241e8dfaef1cdca537fddeee631ae10 with result from task 816 parsl.dataflow.dflow DEBUG: Task 816 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 817 has memoization hash 7c3286b177f0a595720fe2f4d02f742a parsl.dataflow.memoization INFO: Task 817 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 817 try 0 launched on executor multi with executor id 630 parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/153c99c6-13b1-4ca0-9ecc-adecae112988_characterizeImage_898799_35.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/153c99c6-13b1-4ca0-9ecc-adecae112988_characterizeImage_898799_35.stderr parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/280a5be1-369a-485a-8cc7-c005e02be30f_isr_898799_35.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/280a5be1-369a-485a-8cc7-c005e02be30f_isr_898799_35.stderr parsl.dataflow.dflow INFO: Task 555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0eda1c5f19a3ba2dffead0f8da82c1c with result from task 555 parsl.dataflow.dflow DEBUG: Task 555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 556 has memoization hash 97d69c38a15e2abdfc07830e034e09a8 parsl.dataflow.memoization INFO: Task 556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dd6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 631 parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/4fbd67e7-8618-4a5b-b265-0d84ff3cf96f_characterizeImage_898800_42.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/4fbd67e7-8618-4a5b-b265-0d84ff3cf96f_characterizeImage_898800_42.stderr parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/c9f49d0d-b8a8-4650-915b-17d37d8f9115_isr_898800_42.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/c9f49d0d-b8a8-4650-915b-17d37d8f9115_isr_898800_42.stderr parsl.dataflow.dflow INFO: Task 837 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 821d7416a807a19597cc1cccfa4d43ba with result from task 837 parsl.dataflow.dflow DEBUG: Task 837 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 838 has memoization hash e93c0721e26383db889a626d3113b699 parsl.dataflow.memoization INFO: Task 838 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 838 try 0 launched on executor multi with executor id 632 parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/721711bc-4592-4e46-a01f-e7a0ac33c36a_characterizeImage_898798_44.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/721711bc-4592-4e46-a01f-e7a0ac33c36a_characterizeImage_898798_44.stderr parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/adc5886a-c52d-4669-88c7-46407040e294_isr_898798_44.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/adc5886a-c52d-4669-88c7-46407040e294_isr_898798_44.stderr parsl.dataflow.dflow INFO: Task 552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 304d6d35bca2a228e01686ee32de441f with result from task 552 parsl.dataflow.dflow DEBUG: Task 552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 553 has memoization hash 0e787cdc4bf3a5060c3656b362a80215 parsl.dataflow.memoization INFO: Task 553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dd3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 633 parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/066dc6af-9d54-4ce6-a446-f4dc552b10b9_characterizeImage_898800_29.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/066dc6af-9d54-4ce6-a446-f4dc552b10b9_characterizeImage_898800_29.stderr parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/09edc1f6-741e-4f21-97e1-d2cce02f0ab4_isr_898800_29.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/09edc1f6-741e-4f21-97e1-d2cce02f0ab4_isr_898800_29.stderr parsl.dataflow.dflow INFO: Task 558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac26f9915d0b51bbb19939837b4b4200 with result from task 558 parsl.dataflow.dflow DEBUG: Task 558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 559 has memoization hash 4b3e4de70a64a1e6ba008c1b1724b074 parsl.dataflow.memoization INFO: Task 559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dd9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 559 try 0 launched on executor multi with executor id 634 parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/4d15a0c9-4d09-4613-ab8b-6b95fcfca653_characterizeImage_898695_59.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/4d15a0c9-4d09-4613-ab8b-6b95fcfca653_characterizeImage_898695_59.stderr parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/ce0ca861-207d-4b33-a9f9-ff120490c9fd_isr_898695_59.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/ce0ca861-207d-4b33-a9f9-ff120490c9fd_isr_898695_59.stderr parsl.dataflow.dflow INFO: Task 540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6bc1f5cd1d944b8d3ba202b95fd107e with result from task 540 parsl.dataflow.dflow DEBUG: Task 540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 541 has memoization hash 89c047a518e8286f41524ccc30b53807 parsl.dataflow.memoization INFO: Task 541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dc900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 635 parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/24fccec6-1b9f-4118-ac9b-436178150dd5_characterizeImage_898800_20.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/24fccec6-1b9f-4118-ac9b-436178150dd5_characterizeImage_898800_20.stderr parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/76543e45-d206-4b0e-bb25-a9851eb8a310_isr_898800_20.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/76543e45-d206-4b0e-bb25-a9851eb8a310_isr_898800_20.stderr parsl.dataflow.dflow INFO: Task 537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab7669f960cf6a46be320b53c9ceec92 with result from task 537 parsl.dataflow.dflow DEBUG: Task 537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash a4e3f98b4a1a349537d62dcd6ed30404 parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dc5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 636 parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/13a56d53-c439-4415-b6c4-8b2848a05ff6_characterizeImage_898800_32.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/13a56d53-c439-4415-b6c4-8b2848a05ff6_characterizeImage_898800_32.stderr parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/83f138d0-5617-4a8f-b7c9-42f195697f00_isr_898800_32.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/83f138d0-5617-4a8f-b7c9-42f195697f00_isr_898800_32.stderr parsl.dataflow.dflow INFO: Task 831 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de240c94aea65e52d0e43369ae0299d0 with result from task 831 parsl.dataflow.dflow DEBUG: Task 831 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 832 has memoization hash f01dc5a8511875c24f1455847bcc5d8c parsl.dataflow.memoization INFO: Task 832 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 832 try 0 launched on executor multi with executor id 637 parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/8c899deb-43d3-49af-957a-499befb13ffc_characterizeImage_898694_26.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/8c899deb-43d3-49af-957a-499befb13ffc_characterizeImage_898694_26.stderr parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/3c8c2dc4-b7cd-479b-bdda-3e4645d2e91c_isr_898694_26.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/3c8c2dc4-b7cd-479b-bdda-3e4645d2e91c_isr_898694_26.stderr parsl.dataflow.dflow INFO: Task 903 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98da60d14d823e98a29a6939fa90c077 with result from task 903 parsl.dataflow.dflow DEBUG: Task 903 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 904 has memoization hash c33284c3be1e00cc9bff47ea03e94e9b parsl.dataflow.memoization INFO: Task 904 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c443ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 904 try 0 launched on executor multi with executor id 638 parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/565605a4-c6bf-49c2-af54-e90a342b2fe5_characterizeImage_898694_59.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/565605a4-c6bf-49c2-af54-e90a342b2fe5_characterizeImage_898694_59.stderr parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/e690c0b5-b76b-4dba-b762-2d77bd62c06e_isr_898694_59.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/e690c0b5-b76b-4dba-b762-2d77bd62c06e_isr_898694_59.stderr parsl.dataflow.dflow INFO: Task 888 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b91e73cac9294c69d79d319f8094a3b9 with result from task 888 parsl.dataflow.dflow DEBUG: Task 888 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 889 has memoization hash c8627b4fa4fe0cf9f691f3db6172a6f3 parsl.dataflow.memoization INFO: Task 889 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4434c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 889 try 0 launched on executor multi with executor id 639 parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/8ff4aed3-5f64-4427-8e0e-a2449b816468_characterizeImage_898694_52.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/8ff4aed3-5f64-4427-8e0e-a2449b816468_characterizeImage_898694_52.stderr parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/06b0a523-ab6b-43c0-9741-339e4df37853_isr_898694_52.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/06b0a523-ab6b-43c0-9741-339e4df37853_isr_898694_52.stderr parsl.dataflow.dflow INFO: Task 609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ceb38ea41a21440749c72c65e13970a0 with result from task 609 parsl.dataflow.dflow DEBUG: Task 609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 610 has memoization hash 9511ae800e79f794c2f5077cf7f69e69 parsl.dataflow.memoization INFO: Task 610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c511120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 610 try 0 launched on executor multi with executor id 640 parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/78404d00-8116-4989-b1de-a7c334b1dcfe_characterizeImage_898696_60.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/78404d00-8116-4989-b1de-a7c334b1dcfe_characterizeImage_898696_60.stderr parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/16806378-b168-46d4-8e36-71069f35bd80_isr_898696_60.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/16806378-b168-46d4-8e36-71069f35bd80_isr_898696_60.stderr parsl.dataflow.dflow INFO: Task 687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2cdfcffdb7c99ea46330ad0dc3dedfe with result from task 687 parsl.dataflow.dflow DEBUG: Task 687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 688 has memoization hash caf0050e526f99fffabd33bb7f8bc415 parsl.dataflow.memoization INFO: Task 688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3759e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 688 try 0 launched on executor multi with executor id 641 parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/df78ca11-275f-4dfb-907c-70332c874ff0_characterizeImage_898798_29.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/df78ca11-275f-4dfb-907c-70332c874ff0_characterizeImage_898798_29.stderr parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/24626fea-563d-4cc3-97ef-4737e7fe36be_isr_898798_29.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/24626fea-563d-4cc3-97ef-4737e7fe36be_isr_898798_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:46:48 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 891 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 134a81519f7fd7e6e6c0f582be474ac3 with result from task 891 parsl.dataflow.dflow DEBUG: Task 891 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 892 has memoization hash c954498ab32a2ec5b3d30ea5ddbe8da1 parsl.dataflow.memoization INFO: Task 892 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4437e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 892 try 0 launched on executor multi with executor id 642 parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/88975824-e18e-4e1b-89f3-d213b68b672f_characterizeImage_898798_59.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/88975824-e18e-4e1b-89f3-d213b68b672f_characterizeImage_898798_59.stderr parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/4ebebbde-91fa-4525-aa09-3aeda3acb51f_isr_898798_59.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/4ebebbde-91fa-4525-aa09-3aeda3acb51f_isr_898798_59.stderr parsl.dataflow.dflow INFO: Task 846 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 246d1d17d9107257219ed329fe0f3df6 with result from task 846 parsl.dataflow.dflow DEBUG: Task 846 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 847 has memoization hash be386370dda759a42ce0fed9ee5f75a4 parsl.dataflow.memoization INFO: Task 847 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c440900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 847 try 0 launched on executor multi with executor id 643 parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/877730e7-9035-4461-a080-15fbd738c8c1_characterizeImage_898695_52.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/877730e7-9035-4461-a080-15fbd738c8c1_characterizeImage_898695_52.stderr parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/ae6440d1-2994-4b33-8258-56d50cac5c56_isr_898695_52.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/ae6440d1-2994-4b33-8258-56d50cac5c56_isr_898695_52.stderr parsl.dataflow.dflow INFO: Task 732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cffb3cd0506cd36a41c14bb289eebaf5 with result from task 732 parsl.dataflow.dflow DEBUG: Task 732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 733 has memoization hash 80e06e9050d4675ba51804a09b2e06e9 parsl.dataflow.memoization INFO: Task 733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 733 try 0 launched on executor multi with executor id 644 parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/c6efdefd-3dd1-4414-b4d6-ace0736c4137_characterizeImage_898696_8.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/c6efdefd-3dd1-4414-b4d6-ace0736c4137_characterizeImage_898696_8.stderr parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/4e943014-2ceb-40da-950c-7eee1a8bd64a_isr_898696_8.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/4e943014-2ceb-40da-950c-7eee1a8bd64a_isr_898696_8.stderr parsl.dataflow.dflow INFO: Task 546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd2fe20119a08d1df1c0f578e886e2e2 with result from task 546 parsl.dataflow.dflow DEBUG: Task 546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 547 has memoization hash 96900eab0cf83c9a97458a39ed0a9610 parsl.dataflow.memoization INFO: Task 547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dcd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 645 parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/31a6430d-764c-430e-a4c5-2beadf7afd57_characterizeImage_898696_37.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/31a6430d-764c-430e-a4c5-2beadf7afd57_characterizeImage_898696_37.stderr parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/65882770-57ad-489e-a487-731327dd2e9c_isr_898696_37.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/65882770-57ad-489e-a487-731327dd2e9c_isr_898696_37.stderr parsl.dataflow.dflow INFO: Task 726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 722b7aa69261420bf4b40a750daf66cd with result from task 726 parsl.dataflow.dflow DEBUG: Task 726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 727 has memoization hash b18332daa6130cfbec090e98f7f02611 parsl.dataflow.memoization INFO: Task 727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 727 try 0 launched on executor multi with executor id 646 parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/f537cbd5-a625-45ec-a865-8afb2080d283_characterizeImage_898695_57.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/f537cbd5-a625-45ec-a865-8afb2080d283_characterizeImage_898695_57.stderr parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/2489c1e9-947e-4eea-a270-32ac8ee295bc_isr_898695_57.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/2489c1e9-947e-4eea-a270-32ac8ee295bc_isr_898695_57.stderr parsl.dataflow.dflow INFO: Task 1005 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbf6c7e557f9f79e8d7270c49c75e463 with result from task 1005 parsl.dataflow.dflow DEBUG: Task 1005 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1006 has memoization hash f30475a05cbb57856fd3b167428f8bb9 parsl.dataflow.memoization INFO: Task 1006 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2daf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1006 try 0 launched on executor multi with executor id 647 parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/17fe7c0d-7f90-4928-a64d-d2775725ddde_characterizeImage_898695_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/17fe7c0d-7f90-4928-a64d-d2775725ddde_characterizeImage_898695_38.stderr parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/317817ee-e8d5-4c40-8de7-a5445f2f6d1a_isr_898695_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/317817ee-e8d5-4c40-8de7-a5445f2f6d1a_isr_898695_38.stderr parsl.dataflow.dflow INFO: Task 1017 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8abd7631dc389cb69c03b631d389fee1 with result from task 1017 parsl.dataflow.dflow DEBUG: Task 1017 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1018 has memoization hash 06c5f2236cbe5688ce79473009023bba parsl.dataflow.memoization INFO: Task 1018 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2db9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1018 try 0 launched on executor multi with executor id 648 parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/f24380c1-818a-4a62-bcbb-ce3d15491fb0_characterizeImage_898800_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/f24380c1-818a-4a62-bcbb-ce3d15491fb0_characterizeImage_898800_22.stderr parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/3999cc0b-404a-4fc3-b8fa-a0e2fe5c3911_isr_898800_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/3999cc0b-404a-4fc3-b8fa-a0e2fe5c3911_isr_898800_22.stderr parsl.dataflow.dflow INFO: Task 714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36d646e08b04505b4ac107ac1bc1784d with result from task 714 parsl.dataflow.dflow DEBUG: Task 714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 715 has memoization hash 989f698b5c03f4b66aa01d92738a89f4 parsl.dataflow.memoization INFO: Task 715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c377ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 715 try 0 launched on executor multi with executor id 649 parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/f7bc02f4-aac1-46b3-9da4-841a6316922d_characterizeImage_898695_5.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/f7bc02f4-aac1-46b3-9da4-841a6316922d_characterizeImage_898695_5.stderr parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/ad533d91-d02a-4646-9e25-223b381b42b2_isr_898695_5.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/ad533d91-d02a-4646-9e25-223b381b42b2_isr_898695_5.stderr parsl.dataflow.dflow INFO: Task 1011 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9f8e964598696b7812414e242a0fd4e with result from task 1011 parsl.dataflow.dflow DEBUG: Task 1011 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1012 has memoization hash 6ee6d6f080655d9047a0f02698e59752 parsl.dataflow.memoization INFO: Task 1012 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2db560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1012 try 0 launched on executor multi with executor id 650 parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/3720f3ce-b3d4-446f-8571-1c702294eab7_characterizeImage_898696_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/3720f3ce-b3d4-446f-8571-1c702294eab7_characterizeImage_898696_24.stderr parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/d7c86482-ed31-462f-93d2-f623b2c14586_isr_898696_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/d7c86482-ed31-462f-93d2-f623b2c14586_isr_898696_24.stderr parsl.dataflow.dflow INFO: Task 1020 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb3e89ce038244af0773794dbc3ccccd with result from task 1020 parsl.dataflow.dflow DEBUG: Task 1020 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1021 has memoization hash d331d57abc872f7d025c163853d3c1fe parsl.dataflow.memoization INFO: Task 1021 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2dbce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1021 try 0 launched on executor multi with executor id 651 parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/9177541f-17ea-4580-accd-2074ad1b63a3_characterizeImage_898695_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/9177541f-17ea-4580-accd-2074ad1b63a3_characterizeImage_898695_33.stderr parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/64c91e49-24df-4b19-95f0-e2ceffae365b_isr_898695_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/64c91e49-24df-4b19-95f0-e2ceffae365b_isr_898695_33.stderr parsl.dataflow.dflow INFO: Task 711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bc252806c0e5a43a24854e31f2f49af with result from task 711 parsl.dataflow.dflow DEBUG: Task 711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 712 has memoization hash 2940499c8623fcc8707bb8b3fbda0e25 parsl.dataflow.memoization INFO: Task 712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c377ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 712 try 0 launched on executor multi with executor id 652 parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/1325d26e-86d1-45ca-acc8-a3fc81ae3e6f_characterizeImage_898798_5.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/1325d26e-86d1-45ca-acc8-a3fc81ae3e6f_characterizeImage_898798_5.stderr parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/8c72eaab-775b-4cd0-8c1e-73d3b276066b_isr_898798_5.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/8c72eaab-775b-4cd0-8c1e-73d3b276066b_isr_898798_5.stderr parsl.dataflow.dflow INFO: Task 1026 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04695d397fcdc5dba321a3859d2d700a with result from task 1026 parsl.dataflow.dflow DEBUG: Task 1026 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1027 has memoization hash cfd041640aec2f298875f0b63b424e0d parsl.dataflow.memoization INFO: Task 1027 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3280e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1027 try 0 launched on executor multi with executor id 653 parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/13fcdb83-0b8c-4bb4-aa5d-f9134b1eb91f_characterizeImage_898694_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/13fcdb83-0b8c-4bb4-aa5d-f9134b1eb91f_characterizeImage_898694_13.stderr parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/3d840236-f1ba-48f8-a3d1-e314b85e477e_isr_898694_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/3d840236-f1ba-48f8-a3d1-e314b85e477e_isr_898694_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:46:49 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 498dab46db1527dfe6a2bd95f7e1a093 with result from task 723 parsl.dataflow.dflow DEBUG: Task 723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 724 has memoization hash 66e43f2d363b4661723bf3964f3b42b3 parsl.dataflow.memoization INFO: Task 724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 724 try 0 launched on executor multi with executor id 654 parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/4af02fa4-cbbb-437b-b859-e6fa7615728f_characterizeImage_898694_35.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/4af02fa4-cbbb-437b-b859-e6fa7615728f_characterizeImage_898694_35.stderr parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/97995ac4-245f-416b-a47b-c33a58e5a192_isr_898694_35.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/97995ac4-245f-416b-a47b-c33a58e5a192_isr_898694_35.stderr parsl.dataflow.dflow INFO: Task 717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5bb36ae0bcae61ef3ae7a7db7144981 with result from task 717 parsl.dataflow.dflow DEBUG: Task 717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 718 has memoization hash cf261c7f1e6529a20438313511264a18 parsl.dataflow.memoization INFO: Task 718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c377f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 718 try 0 launched on executor multi with executor id 655 parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/eb4fed2f-82dd-4a37-9508-fb1a3e8d479e_characterizeImage_898798_39.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/eb4fed2f-82dd-4a37-9508-fb1a3e8d479e_characterizeImage_898798_39.stderr parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/501cf536-6b25-47f9-a3aa-91a525dda533_isr_898798_39.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/501cf536-6b25-47f9-a3aa-91a525dda533_isr_898798_39.stderr parsl.dataflow.dflow INFO: Task 729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62a59e494ce84e5f958e64d66a38773a with result from task 729 parsl.dataflow.dflow DEBUG: Task 729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 730 has memoization hash 2f09bc5c5ceb60b5e068383360df69bb parsl.dataflow.memoization INFO: Task 730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 730 try 0 launched on executor multi with executor id 656 parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/95f1fb3e-a964-472a-ab76-5d75ee2ec6e8_characterizeImage_898696_50.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/95f1fb3e-a964-472a-ab76-5d75ee2ec6e8_characterizeImage_898696_50.stderr parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/cace4c4d-2909-4540-be48-6de128ad9af2_isr_898696_50.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/cace4c4d-2909-4540-be48-6de128ad9af2_isr_898696_50.stderr parsl.dataflow.dflow INFO: Task 1023 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4047848b1ac29dd3391f3f4e94b50fc with result from task 1023 parsl.dataflow.dflow DEBUG: Task 1023 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1024 has memoization hash 5de56370d8b78a0b569ce347e5a686d9 parsl.dataflow.memoization INFO: Task 1024 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2dbe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1024 try 0 launched on executor multi with executor id 657 parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/92676c99-cc35-4d10-9bcb-29db7391e5a8_characterizeImage_898800_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/92676c99-cc35-4d10-9bcb-29db7391e5a8_characterizeImage_898800_6.stderr parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/39fb7fe2-7378-4f67-a238-cd4404b9ebfc_isr_898800_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/39fb7fe2-7378-4f67-a238-cd4404b9ebfc_isr_898800_6.stderr parsl.dataflow.dflow INFO: Task 720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e482b8ca0ee9ed04e51fbdb79f542c2b with result from task 720 parsl.dataflow.dflow DEBUG: Task 720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 721 has memoization hash ec0635f2e2f8fa384a918d34f6bb5a8f parsl.dataflow.memoization INFO: Task 721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 721 try 0 launched on executor multi with executor id 658 parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/705781c6-5678-404f-b920-8dc5bc0d2ab4_characterizeImage_898696_6.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/705781c6-5678-404f-b920-8dc5bc0d2ab4_characterizeImage_898696_6.stderr parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/940aa248-dc89-4e1e-b84a-682d7e63e3fb_isr_898696_6.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/940aa248-dc89-4e1e-b84a-682d7e63e3fb_isr_898696_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:46:51 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 1014 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84b70a9d19b71adfb3a98be1775af819 with result from task 1014 parsl.dataflow.dflow DEBUG: Task 1014 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1015 has memoization hash 98c247df475aa743c7d92ed9eaf5f5d1 parsl.dataflow.memoization INFO: Task 1015 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c291300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1015 try 0 launched on executor multi with executor id 659 parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/ec930468-25a3-40b8-99e4-143b6ee33931_characterizeImage_898696_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/ec930468-25a3-40b8-99e4-143b6ee33931_characterizeImage_898696_45.stderr parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/dbd3525a-55f0-494e-a27d-a353be4bc48f_isr_898696_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/dbd3525a-55f0-494e-a27d-a353be4bc48f_isr_898696_45.stderr parsl.dataflow.dflow INFO: Task 1038 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 739cbd7a5d58e2596a954c162b86f13d with result from task 1038 parsl.dataflow.dflow DEBUG: Task 1038 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1039 has memoization hash 405f829704b21da5516836501765e574 parsl.dataflow.memoization INFO: Task 1039 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1039 try 0 launched on executor multi with executor id 660 parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/3d3b8cee-0e42-4830-bbbe-07571de9a323_characterizeImage_898800_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/3d3b8cee-0e42-4830-bbbe-07571de9a323_characterizeImage_898800_33.stderr parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/95efd55a-6361-468b-a475-d41d582d9ade_isr_898800_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/95efd55a-6361-468b-a475-d41d582d9ade_isr_898800_33.stderr parsl.dataflow.dflow INFO: Task 1032 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eeb49c5f1bf4fa7baabb81f5ce555aee with result from task 1032 parsl.dataflow.dflow DEBUG: Task 1032 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1033 has memoization hash 7ffc3addcd1ca03a85bad88fadfb6d2f parsl.dataflow.memoization INFO: Task 1033 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1033 try 0 launched on executor multi with executor id 661 parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/b4850d3d-715c-4d24-9000-9e47d05da923_characterizeImage_898798_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/b4850d3d-715c-4d24-9000-9e47d05da923_characterizeImage_898798_26.stderr parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/6d6dc98b-2ec8-40eb-afb9-c6fbc637c7dd_isr_898798_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/6d6dc98b-2ec8-40eb-afb9-c6fbc637c7dd_isr_898798_26.stderr parsl.dataflow.dflow INFO: Task 1035 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8350707472b0085c99611d08363fbda2 with result from task 1035 parsl.dataflow.dflow DEBUG: Task 1035 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1036 has memoization hash b0871e81032c443d811cc7355f982f04 parsl.dataflow.memoization INFO: Task 1036 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1036 try 0 launched on executor multi with executor id 662 parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/8fcf5d18-9ebc-419f-9aa5-de271b39895c_characterizeImage_898694_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/8fcf5d18-9ebc-419f-9aa5-de271b39895c_characterizeImage_898694_33.stderr parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/880f63c9-3547-4b22-a543-c72da297c625_isr_898694_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/880f63c9-3547-4b22-a543-c72da297c625_isr_898694_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:46:52 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 1029 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9fab1858216aeb7a49a77d9c8567249 with result from task 1029 parsl.dataflow.dflow DEBUG: Task 1029 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1030 has memoization hash ca17b060ce9bf8cafc1fdc39c43368ff parsl.dataflow.memoization INFO: Task 1030 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1030 try 0 launched on executor multi with executor id 663 parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/e4ac50b5-703e-4586-9f02-455fb7be6f7d_characterizeImage_898798_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/e4ac50b5-703e-4586-9f02-455fb7be6f7d_characterizeImage_898798_14.stderr parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/09646f2b-0617-4d05-9dc5-c71fd0716f24_isr_898798_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/09646f2b-0617-4d05-9dc5-c71fd0716f24_isr_898798_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:46:53 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:46:54 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:46:55 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 1047 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed7af9acd016eb5855892b8f18440ff0 with result from task 1047 parsl.dataflow.dflow DEBUG: Task 1047 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1048 has memoization hash 2fb7e5cd388c9d2e838752384499453c parsl.dataflow.memoization INFO: Task 1048 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1048 try 0 launched on executor multi with executor id 664 parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/46c31303-98e4-42ce-a32f-12c1908a604e_characterizeImage_898696_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/46c31303-98e4-42ce-a32f-12c1908a604e_characterizeImage_898696_9.stderr parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/98986982-3857-4ae7-9f8c-e2129b5b182e_isr_898696_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/98986982-3857-4ae7-9f8c-e2129b5b182e_isr_898696_9.stderr parsl.dataflow.dflow INFO: Task 1074 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b521dd290c498a77b87467ce2133bd45 with result from task 1074 parsl.dataflow.dflow DEBUG: Task 1074 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1075 has memoization hash e342e6697b420d831749ba5fd274c4a9 parsl.dataflow.memoization INFO: Task 1075 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c329b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1075 try 0 launched on executor multi with executor id 665 parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/1795edc5-ea45-441b-9448-94efa20a8a89_characterizeImage_898695_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/1795edc5-ea45-441b-9448-94efa20a8a89_characterizeImage_898695_15.stderr parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/4f1ce603-c88f-400b-a4bf-fcaf9cbabf46_isr_898695_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/4f1ce603-c88f-400b-a4bf-fcaf9cbabf46_isr_898695_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:46:56 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 1059 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dee01d0ef34baadc219adf0b4067a734 with result from task 1059 parsl.dataflow.dflow DEBUG: Task 1059 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1060 has memoization hash 3e961373062c64b635363c50c3a58406 parsl.dataflow.memoization INFO: Task 1060 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c328a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1060 try 0 launched on executor multi with executor id 666 parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/96906a5e-0eca-4503-963d-cc645fe2cbf8_characterizeImage_898696_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/96906a5e-0eca-4503-963d-cc645fe2cbf8_characterizeImage_898696_61.stderr parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/ec724237-1bd6-4e96-aa1e-87f376aa0abb_isr_898696_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/ec724237-1bd6-4e96-aa1e-87f376aa0abb_isr_898696_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:46:57 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 1107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 904cbc93d29ed39d5bd63c2ed13141c2 with result from task 1107 parsl.dataflow.dflow DEBUG: Task 1107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1108 has memoization hash fd29ca99ae392b91ffef6e4258f8bdf5 parsl.dataflow.memoization INFO: Task 1108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3519e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1108 try 0 launched on executor multi with executor id 667 parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/ca2dbbaf-44e1-4aa6-a926-cee55871bb36_characterizeImage_898695_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/ca2dbbaf-44e1-4aa6-a926-cee55871bb36_characterizeImage_898695_53.stderr parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/3c6c34a9-b96c-4030-be87-985ea9b2ee63_isr_898695_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/3c6c34a9-b96c-4030-be87-985ea9b2ee63_isr_898695_53.stderr parsl.dataflow.dflow INFO: Task 1113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 914298d58f8f0fdd563dc4876acfc0e3 with result from task 1113 parsl.dataflow.dflow DEBUG: Task 1113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1114 has memoization hash 18b621225e636b3978c6e5b95a799da9 parsl.dataflow.memoization INFO: Task 1114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c352020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1114 try 0 launched on executor multi with executor id 668 parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/2ede117f-f90a-4e69-b281-7d238015d6a1_characterizeImage_898799_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/2ede117f-f90a-4e69-b281-7d238015d6a1_characterizeImage_898799_20.stderr parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/9b6ebefd-22c3-4ef7-92e5-6ffd3528381c_isr_898799_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/9b6ebefd-22c3-4ef7-92e5-6ffd3528381c_isr_898799_20.stderr parsl.dataflow.dflow INFO: Task 1110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c77733e9304c87dda0f359a9ae73509e with result from task 1110 parsl.dataflow.dflow DEBUG: Task 1110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1111 has memoization hash cf5774f2f7f0b2b39906e6f235bb76be parsl.dataflow.memoization INFO: Task 1111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c351d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1111 try 0 launched on executor multi with executor id 669 parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/91cb77ca-d098-4e3d-a37d-ded2a6fc4fd5_characterizeImage_898696_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/91cb77ca-d098-4e3d-a37d-ded2a6fc4fd5_characterizeImage_898696_56.stderr parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/7cd34713-454c-4633-8a5d-bc671360be42_isr_898696_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/7cd34713-454c-4633-8a5d-bc671360be42_isr_898696_56.stderr parsl.dataflow.dflow INFO: Task 1044 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df41f479eeb49a7364227c822ab92d6f with result from task 1044 parsl.dataflow.dflow DEBUG: Task 1044 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1045 has memoization hash e3f92e664aa2baa96cb9104c0694f336 parsl.dataflow.memoization INFO: Task 1045 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1045 try 0 launched on executor multi with executor id 670 parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/b6ed734c-f6ca-449f-a517-ab9935c2766b_characterizeImage_898800_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/b6ed734c-f6ca-449f-a517-ab9935c2766b_characterizeImage_898800_8.stderr parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/96c5124e-15ad-4874-8036-7e4f67ec23c6_isr_898800_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/96c5124e-15ad-4874-8036-7e4f67ec23c6_isr_898800_8.stderr parsl.dataflow.dflow INFO: Task 1095 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30b97032b67a365fc215620344d4ad93 with result from task 1095 parsl.dataflow.dflow DEBUG: Task 1095 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1096 has memoization hash bc60ab7e53278bc689c38ffea5849e69 parsl.dataflow.memoization INFO: Task 1096 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c350d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1096 try 0 launched on executor multi with executor id 671 parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/977970fa-9fd4-42ee-977b-81128df035ac_characterizeImage_898798_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/977970fa-9fd4-42ee-977b-81128df035ac_characterizeImage_898798_40.stderr parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/afa20b57-8fb5-4d22-9428-380c3a479698_isr_898798_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/afa20b57-8fb5-4d22-9428-380c3a479698_isr_898798_40.stderr parsl.dataflow.dflow INFO: Task 1101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 185034f07822fb0a7163df0086405288 with result from task 1101 parsl.dataflow.dflow DEBUG: Task 1101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1102 has memoization hash cdbfca88db65fc0f6abcccfdc3ada566 parsl.dataflow.memoization INFO: Task 1102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3513a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1102 try 0 launched on executor multi with executor id 672 parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/5fd2d5e5-5cb0-4a54-87b3-16a8ebfca6fa_characterizeImage_898695_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/5fd2d5e5-5cb0-4a54-87b3-16a8ebfca6fa_characterizeImage_898695_3.stderr parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/61142537-2aa0-41f8-b75d-bcf562f7980e_isr_898695_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/61142537-2aa0-41f8-b75d-bcf562f7980e_isr_898695_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:46:58 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 1092 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29c51cdc7d2d76dd6a063aea498dbc69 with result from task 1092 parsl.dataflow.dflow DEBUG: Task 1092 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1093 has memoization hash 976612c3e713d07af1648344ea425d7a parsl.dataflow.memoization INFO: Task 1093 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c350a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1093 try 0 launched on executor multi with executor id 673 parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/b5db42d4-bde8-40d1-8b21-57ac01a24056_characterizeImage_898694_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/b5db42d4-bde8-40d1-8b21-57ac01a24056_characterizeImage_898694_12.stderr parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/82decbe3-a1b8-40fc-ad05-ded92deea7cc_isr_898694_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/82decbe3-a1b8-40fc-ad05-ded92deea7cc_isr_898694_12.stderr parsl.dataflow.dflow INFO: Task 1098 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8d2712c4cba476572f9afd1dbc60d2e with result from task 1098 parsl.dataflow.dflow DEBUG: Task 1098 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1099 has memoization hash 3b41640edb91a52d9a2b04d6a49c9568 parsl.dataflow.memoization INFO: Task 1099 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c351080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1099 try 0 launched on executor multi with executor id 674 parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/5be748ac-adbd-40fb-a77a-e6d235d47721_characterizeImage_898800_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/5be748ac-adbd-40fb-a77a-e6d235d47721_characterizeImage_898800_50.stderr parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/f1fa8264-3fee-4c14-b46f-873c133f5f00_isr_898800_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/f1fa8264-3fee-4c14-b46f-873c133f5f00_isr_898800_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:00 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 1089 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5efb92a40357fc83da13902075740a47 with result from task 1089 parsl.dataflow.dflow DEBUG: Task 1089 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1090 has memoization hash 1e22cd787c9e8b03ea6cf63bd27f909c parsl.dataflow.memoization INFO: Task 1090 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c350720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1090 try 0 launched on executor multi with executor id 675 parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/9de6b8ac-60ff-413f-b6d5-5ff4cbabe482_characterizeImage_898696_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/9de6b8ac-60ff-413f-b6d5-5ff4cbabe482_characterizeImage_898696_19.stderr parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/3d573d67-cfa2-4186-8a77-7282c927638b_isr_898696_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/3d573d67-cfa2-4186-8a77-7282c927638b_isr_898696_19.stderr parsl.dataflow.dflow INFO: Task 1104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06e823adbfc608f08122fe563dc54cdc with result from task 1104 parsl.dataflow.dflow DEBUG: Task 1104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1105 has memoization hash 413f031c4d16b405fddbb1a965ba696e parsl.dataflow.memoization INFO: Task 1105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3516c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1105 try 0 launched on executor multi with executor id 676 parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/3b7ee91d-fbb4-4a34-95ad-d8f2f428f744_characterizeImage_898798_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/3b7ee91d-fbb4-4a34-95ad-d8f2f428f744_characterizeImage_898798_43.stderr parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/19d906ea-7077-40d3-94e8-d4af7b15814b_isr_898798_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/19d906ea-7077-40d3-94e8-d4af7b15814b_isr_898798_43.stderr parsl.dataflow.dflow INFO: Task 765 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 491ca97a8e2edad31a5694eb1eb6d8d7 with result from task 765 parsl.dataflow.dflow DEBUG: Task 765 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 766 has memoization hash 74fbd11f427a0d15ff2e4a8cb33a8230 parsl.dataflow.memoization INFO: Task 766 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 766 try 0 launched on executor multi with executor id 677 parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/d923957f-e169-470c-92ea-e9457d194cfc_characterizeImage_898695_32.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/d923957f-e169-470c-92ea-e9457d194cfc_characterizeImage_898695_32.stderr parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/22912694-60d5-44e2-9319-ee50f0dad658_isr_898695_32.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/22912694-60d5-44e2-9319-ee50f0dad658_isr_898695_32.stderr parsl.dataflow.dflow INFO: Task 783 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b74ad18ccfa241c1a297507976655c13 with result from task 783 parsl.dataflow.dflow DEBUG: Task 783 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 784 has memoization hash 36d944b72292755f4332cbd7202482fb parsl.dataflow.memoization INFO: Task 784 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3bb100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 784 try 0 launched on executor multi with executor id 678 parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/77515ba1-4685-4b79-b3fd-01a10dd7da13_characterizeImage_898696_32.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/77515ba1-4685-4b79-b3fd-01a10dd7da13_characterizeImage_898696_32.stderr parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/75630e31-b7fd-4960-9d24-7972f3439a70_isr_898696_32.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/75630e31-b7fd-4960-9d24-7972f3439a70_isr_898696_32.stderr parsl.dataflow.dflow INFO: Task 798 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38c887e918a8b32a37cc385e0fbe308e with result from task 798 parsl.dataflow.dflow DEBUG: Task 798 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 799 has memoization hash 08ab8fb4c73f2340a2ce54baac0f28b7 parsl.dataflow.memoization INFO: Task 799 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 799 try 0 launched on executor multi with executor id 679 parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/68ca4089-247f-463f-831a-a6c66b35297a_characterizeImage_898800_19.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/68ca4089-247f-463f-831a-a6c66b35297a_characterizeImage_898800_19.stderr parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/1e771db2-bde5-4cf8-8db9-59c6e9df85f8_isr_898800_19.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/1e771db2-bde5-4cf8-8db9-59c6e9df85f8_isr_898800_19.stderr parsl.dataflow.dflow INFO: Task 1050 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f467a3d1a5a25d6427e2a83691ea7047 with result from task 1050 parsl.dataflow.dflow DEBUG: Task 1050 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1051 has memoization hash b2fd536113ba461af81c3a7f5e9d6feb parsl.dataflow.memoization INFO: Task 1051 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1051 try 0 launched on executor multi with executor id 680 parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/758c13da-3894-4988-9dda-f4358093d5d4_characterizeImage_898800_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/758c13da-3894-4988-9dda-f4358093d5d4_characterizeImage_898800_9.stderr parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/d4fc5835-d031-4b21-a3c0-c800b7503e61_isr_898800_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/d4fc5835-d031-4b21-a3c0-c800b7503e61_isr_898800_9.stderr parsl.dataflow.dflow INFO: Task 1062 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2feb32e37bca754ac4cfb23b57a45a8 with result from task 1062 parsl.dataflow.dflow DEBUG: Task 1062 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1063 has memoization hash fa1c2006897f3a52b2eb6024bfc1afb9 parsl.dataflow.memoization INFO: Task 1063 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c328e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1063 try 0 launched on executor multi with executor id 681 parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/dd26452c-0598-445c-b2af-262387565937_characterizeImage_898695_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/dd26452c-0598-445c-b2af-262387565937_characterizeImage_898695_9.stderr parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/267a88bd-3492-496c-b04d-cb88f5756c3d_isr_898695_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/267a88bd-3492-496c-b04d-cb88f5756c3d_isr_898695_9.stderr parsl.dataflow.dflow INFO: Task 1041 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65a807dfc99d8848e4abc63691577e71 with result from task 1041 parsl.dataflow.dflow DEBUG: Task 1041 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1042 has memoization hash 70baa38865c63c6a162c33043362febc parsl.dataflow.memoization INFO: Task 1042 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1042 try 0 launched on executor multi with executor id 682 parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/5a750542-aad2-4c5b-9654-8ce69a9633e4_characterizeImage_898798_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/5a750542-aad2-4c5b-9654-8ce69a9633e4_characterizeImage_898798_27.stderr parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/167e2666-4b68-4b1a-8f71-85edc21cb6ab_isr_898798_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/167e2666-4b68-4b1a-8f71-85edc21cb6ab_isr_898798_27.stderr parsl.dataflow.dflow INFO: Task 1086 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa47700a03f618dcf1fb2d255b0b0041 with result from task 1086 parsl.dataflow.dflow DEBUG: Task 1086 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1087 has memoization hash 32ccebbcb10c5dbe283275dbbe2e9165 parsl.dataflow.memoization INFO: Task 1087 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3504a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1087 try 0 launched on executor multi with executor id 683 parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/e8e7027e-0930-465c-b516-ae76c660cd30_characterizeImage_898800_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/e8e7027e-0930-465c-b516-ae76c660cd30_characterizeImage_898800_16.stderr parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/df0f69c1-5256-4757-807e-cfd544b104d9_isr_898800_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/df0f69c1-5256-4757-807e-cfd544b104d9_isr_898800_16.stderr parsl.dataflow.dflow INFO: Task 780 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ecc893e023f8f6bdb3e99e150c83dad with result from task 780 parsl.dataflow.dflow DEBUG: Task 780 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 781 has memoization hash f4df04e63a704bed3bf64235164cdded parsl.dataflow.memoization INFO: Task 781 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 781 try 0 launched on executor multi with executor id 684 parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/f9decf9f-9926-4347-b72c-1d9e3d47aad7_characterizeImage_898800_28.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/f9decf9f-9926-4347-b72c-1d9e3d47aad7_characterizeImage_898800_28.stderr parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/13fab25c-8b2d-4b86-9b2c-3f8b7d768958_isr_898800_28.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/13fab25c-8b2d-4b86-9b2c-3f8b7d768958_isr_898800_28.stderr parsl.dataflow.dflow INFO: Task 801 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7b4c641405d2ce0962d863b3b0a99ab with result from task 801 parsl.dataflow.dflow DEBUG: Task 801 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 802 has memoization hash 73a8541a0c28358c8cb824dd22b0b426 parsl.dataflow.memoization INFO: Task 802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 802 try 0 launched on executor multi with executor id 685 parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/b3efbc14-5c18-4e2b-9491-e0294b592d00_characterizeImage_898694_32.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/b3efbc14-5c18-4e2b-9491-e0294b592d00_characterizeImage_898694_32.stderr parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/7c10280c-45f9-4b24-a9b9-fffc88e35481_isr_898694_32.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/7c10280c-45f9-4b24-a9b9-fffc88e35481_isr_898694_32.stderr parsl.dataflow.dflow INFO: Task 1071 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94cc2cc4dae02319eb90d6e82da2bce4 with result from task 1071 parsl.dataflow.dflow DEBUG: Task 1071 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1072 has memoization hash e2dc9182bdb0620a8bc03c2079467b50 parsl.dataflow.memoization INFO: Task 1072 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c329800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1072 try 0 launched on executor multi with executor id 686 parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/3457e19d-e614-4fff-9940-e110e69df94f_characterizeImage_898799_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/3457e19d-e614-4fff-9940-e110e69df94f_characterizeImage_898799_43.stderr parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/a9794ef2-5a2b-4a12-8f81-3bee8acebef6_isr_898799_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/a9794ef2-5a2b-4a12-8f81-3bee8acebef6_isr_898799_43.stderr parsl.dataflow.dflow INFO: Task 1080 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 557ed074d027670f715cf4a1601bac2c with result from task 1080 parsl.dataflow.dflow DEBUG: Task 1080 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1081 has memoization hash a93fa3a7118ad475dc1948b41fafb175 parsl.dataflow.memoization INFO: Task 1081 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1081 try 0 launched on executor multi with executor id 687 parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/53e233bf-39f2-44f2-816b-76732127389d_characterizeImage_898695_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/53e233bf-39f2-44f2-816b-76732127389d_characterizeImage_898695_23.stderr parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/3026279c-3fb0-4e26-928d-fe9c223afe86_isr_898695_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/3026279c-3fb0-4e26-928d-fe9c223afe86_isr_898695_23.stderr parsl.dataflow.dflow INFO: Task 768 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53d7093c65826b579b39f96444e0c1c2 with result from task 768 parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:01 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow DEBUG: Task 768 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 769 has memoization hash edc57ca699368f7750566381a2edec44 parsl.dataflow.memoization INFO: Task 769 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 769 try 0 launched on executor multi with executor id 688 parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/d1c24002-b728-418a-8c26-05f05da239ec_characterizeImage_898800_10.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/d1c24002-b728-418a-8c26-05f05da239ec_characterizeImage_898800_10.stderr parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/41b47ea8-0901-4bd7-9dee-542ea4121eb7_isr_898800_10.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/41b47ea8-0901-4bd7-9dee-542ea4121eb7_isr_898800_10.stderr parsl.dataflow.dflow INFO: Task 777 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69ba715227023c8b6911a853033280ba with result from task 777 parsl.dataflow.dflow DEBUG: Task 777 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 778 has memoization hash ca93ba524a21724505a8b9edefa7466b parsl.dataflow.memoization INFO: Task 778 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 778 try 0 launched on executor multi with executor id 689 parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/778e0cd9-e824-499a-a43b-2f00a433a97f_characterizeImage_898800_13.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/778e0cd9-e824-499a-a43b-2f00a433a97f_characterizeImage_898800_13.stderr parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/1d88ffb0-bbbc-41f4-b7f9-e1788dd925de_isr_898800_13.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/1d88ffb0-bbbc-41f4-b7f9-e1788dd925de_isr_898800_13.stderr parsl.dataflow.dflow INFO: Task 786 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21338850196b658cea5dc3c4bb755389 with result from task 786 parsl.dataflow.dflow DEBUG: Task 786 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 787 has memoization hash 66b1a60f456d0b4f89a25acafb32f92c parsl.dataflow.memoization INFO: Task 787 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3bb740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 787 try 0 launched on executor multi with executor id 690 parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/153275a5-848e-404f-85fb-fdb2d5f5e3d7_characterizeImage_898799_16.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/153275a5-848e-404f-85fb-fdb2d5f5e3d7_characterizeImage_898799_16.stderr parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/4b4ecef7-fe87-458e-a25b-036ce1427113_isr_898799_16.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/4b4ecef7-fe87-458e-a25b-036ce1427113_isr_898799_16.stderr parsl.dataflow.dflow INFO: Task 1056 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02219285c93082f33cc930c29d28e664 with result from task 1056 parsl.dataflow.dflow DEBUG: Task 1056 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1057 has memoization hash dedd27e2c7e264f75e50624d4bc18e01 parsl.dataflow.memoization INFO: Task 1057 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3285e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1057 try 0 launched on executor multi with executor id 691 parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/c1107c1d-285a-440f-8802-b6685b40a352_characterizeImage_898798_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/c1107c1d-285a-440f-8802-b6685b40a352_characterizeImage_898798_48.stderr parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/20a5a27c-f9c9-4f18-83f9-1dfcb18b06df_isr_898798_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/20a5a27c-f9c9-4f18-83f9-1dfcb18b06df_isr_898798_48.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 244 try 0 failed parsl.dataflow.dflow INFO: Task 244 marked for retry parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/a91461ad-3730-4ccd-81a7-8a82038a38a0_characterizeImage_898799_2.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/a91461ad-3730-4ccd-81a7-8a82038a38a0_characterizeImage_898799_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash aaefa86c8314ae4c1be5230d360abe77 parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d608ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 1 launched on executor multi with executor id 692 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/a91461ad-3730-4ccd-81a7-8a82038a38a0_characterizeImage_898799_2.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/a91461ad-3730-4ccd-81a7-8a82038a38a0_characterizeImage_898799_2.stderr parsl.dataflow.dflow INFO: Task 789 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a12c4c5cb399ac9249e68f515a43257 with result from task 789 parsl.dataflow.dflow DEBUG: Task 789 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 790 has memoization hash 2670685054ccc60f57cb296510a66110 parsl.dataflow.memoization INFO: Task 790 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3bbb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 790 try 0 launched on executor multi with executor id 693 parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/92ff6e5e-916a-42f8-8251-f0cd6370dffc_characterizeImage_898799_61.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/92ff6e5e-916a-42f8-8251-f0cd6370dffc_characterizeImage_898799_61.stderr parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/b42611e2-287a-4baa-b856-f2975e06b01d_isr_898799_61.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/b42611e2-287a-4baa-b856-f2975e06b01d_isr_898799_61.stderr parsl.dataflow.dflow INFO: Task 807 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c85a451dc474193bc4cb696ef9948f95 with result from task 807 parsl.dataflow.dflow DEBUG: Task 807 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 808 has memoization hash f67f93d412a6eb955eaa2f10c1710b4a parsl.dataflow.memoization INFO: Task 808 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 808 try 0 launched on executor multi with executor id 694 parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/03215a29-38e3-4542-b499-b8d432f0d075_characterizeImage_898799_55.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/03215a29-38e3-4542-b499-b8d432f0d075_characterizeImage_898799_55.stderr parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/fa1b9937-03fe-4fe2-a634-0199ccb50687_isr_898799_55.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/fa1b9937-03fe-4fe2-a634-0199ccb50687_isr_898799_55.stderr parsl.dataflow.dflow INFO: Task 1068 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 827ca6d402c80937d49dd2bee32d6898 with result from task 1068 parsl.dataflow.dflow DEBUG: Task 1068 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1069 has memoization hash 4aa323695e5a488d4aa492bbbaf08ccc parsl.dataflow.memoization INFO: Task 1069 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3294e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1069 try 0 launched on executor multi with executor id 695 parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/dcd867b3-0992-4a38-9783-66af55b68770_characterizeImage_898696_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/dcd867b3-0992-4a38-9783-66af55b68770_characterizeImage_898696_11.stderr parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/202eb645-60e1-4530-99f0-987ec51af627_isr_898696_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/202eb645-60e1-4530-99f0-987ec51af627_isr_898696_11.stderr parsl.dataflow.dflow INFO: Task 810 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6df8c030a0b1b62db25263cdf81ec8c8 with result from task 810 parsl.dataflow.dflow DEBUG: Task 810 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 811 has memoization hash 23c257baa6422c1a662a7c1d54d5f177 parsl.dataflow.memoization INFO: Task 811 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 811 try 0 launched on executor multi with executor id 696 parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/8541934e-01ba-4fba-a5fd-0ad3ff8f4808_characterizeImage_898799_17.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/8541934e-01ba-4fba-a5fd-0ad3ff8f4808_characterizeImage_898799_17.stderr parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/0833e567-18e2-4f4e-b81b-58eb68ee0b67_isr_898799_17.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/0833e567-18e2-4f4e-b81b-58eb68ee0b67_isr_898799_17.stderr parsl.dataflow.dflow INFO: Task 1053 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5244404b80bea21db020634059c7054 with result from task 1053 parsl.dataflow.dflow DEBUG: Task 1053 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1054 has memoization hash 99383185b307567987c2f88af3c6ec3b parsl.dataflow.memoization INFO: Task 1054 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1054 try 0 launched on executor multi with executor id 697 parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/d482ea6f-1bb7-4b7d-8534-bc1fe5ea4cd1_characterizeImage_898800_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/d482ea6f-1bb7-4b7d-8534-bc1fe5ea4cd1_characterizeImage_898800_55.stderr parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/3399a077-2074-4bf4-a72a-9c1dcba947f8_isr_898800_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/3399a077-2074-4bf4-a72a-9c1dcba947f8_isr_898800_55.stderr parsl.dataflow.dflow INFO: Task 792 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2b856db7679d874cc1b88d9ef7ee993 with result from task 792 parsl.dataflow.dflow DEBUG: Task 792 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 793 has memoization hash 427f57779a473800837c7632ef28ff90 parsl.dataflow.memoization INFO: Task 793 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3bbe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 793 try 0 launched on executor multi with executor id 698 parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/7718c854-b248-46d2-a245-db7394b3be5e_characterizeImage_898800_44.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/7718c854-b248-46d2-a245-db7394b3be5e_characterizeImage_898800_44.stderr parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/e23ec1b5-0a07-455d-9fd5-634e715fea86_isr_898800_44.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/e23ec1b5-0a07-455d-9fd5-634e715fea86_isr_898800_44.stderr parsl.dataflow.dflow INFO: Task 771 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4f428f15622919609a142218bba0ba5 with result from task 771 parsl.dataflow.dflow DEBUG: Task 771 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 772 has memoization hash 3b679c08d61608d76981c3ded207952b parsl.dataflow.memoization INFO: Task 772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 772 try 0 launched on executor multi with executor id 699 parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/761e1ce1-11ef-406b-9922-fc27fa7c8ff7_characterizeImage_898695_7.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/761e1ce1-11ef-406b-9922-fc27fa7c8ff7_characterizeImage_898695_7.stderr parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/01b6fb62-742f-4824-ba8f-843fdad73460_isr_898695_7.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/01b6fb62-742f-4824-ba8f-843fdad73460_isr_898695_7.stderr parsl.dataflow.dflow INFO: Task 1083 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cbf78a8569d3cd7531edc5c07eafad0 with result from task 1083 parsl.dataflow.dflow DEBUG: Task 1083 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1084 has memoization hash 4666dce1da757b44e18bb3346b7e556c parsl.dataflow.memoization INFO: Task 1084 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1084 try 0 launched on executor multi with executor id 700 parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/3b44af96-4c82-455e-822e-d93ffa466e67_characterizeImage_898695_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/3b44af96-4c82-455e-822e-d93ffa466e67_characterizeImage_898695_37.stderr parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/f2c015a5-f6c6-40b1-895b-1bf7538c7202_isr_898695_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/f2c015a5-f6c6-40b1-895b-1bf7538c7202_isr_898695_37.stderr parsl.dataflow.dflow INFO: Task 774 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79c0c2dfd064745efa9e76d427f5fdcc with result from task 774 parsl.dataflow.dflow DEBUG: Task 774 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 775 has memoization hash 27df080f51b282c70d56bd29fab5b7a1 parsl.dataflow.memoization INFO: Task 775 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 775 try 0 launched on executor multi with executor id 701 parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/46d8fcbc-bd6f-429c-8be5-f8940033e295_characterizeImage_898694_4.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/46d8fcbc-bd6f-429c-8be5-f8940033e295_characterizeImage_898694_4.stderr parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/3c6995ab-224c-4226-bb91-c755fd377c45_isr_898694_4.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/3c6995ab-224c-4226-bb91-c755fd377c45_isr_898694_4.stderr parsl.dataflow.dflow INFO: Task 852 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d4f96b42ee8bd372e508af48a5550f0 with result from task 852 parsl.dataflow.dflow DEBUG: Task 852 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 853 has memoization hash 528f541be1f8cacb883fff78f2a50f63 parsl.dataflow.memoization INFO: Task 853 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c440f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 853 try 0 launched on executor multi with executor id 702 parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/be4d7d79-191a-439b-a951-bbc440d0280f_characterizeImage_898694_8.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/be4d7d79-191a-439b-a951-bbc440d0280f_characterizeImage_898694_8.stderr parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/6db1ffd4-daf9-4063-bd0a-5cc470744ae9_isr_898694_8.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/6db1ffd4-daf9-4063-bd0a-5cc470744ae9_isr_898694_8.stderr parsl.dataflow.dflow INFO: Task 855 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 708bb8e0c4909af25b3da6d17c287c01 with result from task 855 parsl.dataflow.dflow DEBUG: Task 855 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 856 has memoization hash 4d4a03d116e8e5bb24b6d4e1ab8099db parsl.dataflow.memoization INFO: Task 856 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c441260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 856 try 0 launched on executor multi with executor id 703 parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/48405b23-b761-43cc-a39a-298193b8114a_characterizeImage_898694_51.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/48405b23-b761-43cc-a39a-298193b8114a_characterizeImage_898694_51.stderr parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/65262dfb-3d51-4170-b32b-5dfc40f4ea87_isr_898694_51.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/65262dfb-3d51-4170-b32b-5dfc40f4ea87_isr_898694_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:02 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 795 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 157e2707f5f549fd2011f376297d4f14 with result from task 795 parsl.dataflow.dflow DEBUG: Task 795 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 796 has memoization hash 058d09ae509dcbd9f5b038b7e5eef197 parsl.dataflow.memoization INFO: Task 796 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3bbf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 796 try 0 launched on executor multi with executor id 704 parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/affa5b59-d1c0-4830-a90d-ef67ec6d5796_characterizeImage_898798_36.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/affa5b59-d1c0-4830-a90d-ef67ec6d5796_characterizeImage_898798_36.stderr parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/d7f97609-9225-4aa5-9bf8-359c0eb4385d_isr_898798_36.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/d7f97609-9225-4aa5-9bf8-359c0eb4385d_isr_898798_36.stderr parsl.dataflow.dflow INFO: Task 1065 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02cf6bb00c5b995d36b324f013dd5ffa with result from task 1065 parsl.dataflow.dflow DEBUG: Task 1065 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1066 has memoization hash 9ec77fd71990744c36a3d147ec99a4e0 parsl.dataflow.memoization INFO: Task 1066 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c329120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1066 try 0 launched on executor multi with executor id 705 parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/550cc592-e72e-4cda-955a-ef527abbb87b_characterizeImage_898798_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/550cc592-e72e-4cda-955a-ef527abbb87b_characterizeImage_898798_37.stderr parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/05f2e988-e934-4a41-996e-225f96d01d9f_isr_898798_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/05f2e988-e934-4a41-996e-225f96d01d9f_isr_898798_37.stderr parsl.dataflow.dflow INFO: Task 804 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 825cf1c4f7838ec4ec35bd78ece16f4b with result from task 804 parsl.dataflow.dflow DEBUG: Task 804 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 805 has memoization hash 3b266dcd65d9755ae9d2326872d7b26b parsl.dataflow.memoization INFO: Task 805 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 805 try 0 launched on executor multi with executor id 706 parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/5c9daef3-a5cb-4d5f-bed3-70fd2d7569cb_characterizeImage_898799_54.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/5c9daef3-a5cb-4d5f-bed3-70fd2d7569cb_characterizeImage_898799_54.stderr parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/6088d40c-3ed0-4a24-b7e4-088088cbfc8b_isr_898799_54.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/6088d40c-3ed0-4a24-b7e4-088088cbfc8b_isr_898799_54.stderr parsl.dataflow.dflow INFO: Task 1077 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d877b16c2bed5a7ace5089d2665433c9 with result from task 1077 parsl.dataflow.dflow DEBUG: Task 1077 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1078 has memoization hash f065869e58ceec1d7993ba69a9eeab3e parsl.dataflow.memoization INFO: Task 1078 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c329e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1078 try 0 launched on executor multi with executor id 707 parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/57d27afa-c37e-4f64-8797-f29d3b5ee3e9_characterizeImage_898696_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/57d27afa-c37e-4f64-8797-f29d3b5ee3e9_characterizeImage_898696_58.stderr parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/c25bfb1a-851b-43fd-a73f-d543ae8c97a7_isr_898696_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/c25bfb1a-851b-43fd-a73f-d543ae8c97a7_isr_898696_58.stderr parsl.dataflow.dflow INFO: Task 948 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91681ccd679d5700f7468c879400c7ed with result from task 948 parsl.dataflow.dflow DEBUG: Task 948 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 949 has memoization hash 17940b7b86211d58e6922cf4d68cf1ff parsl.dataflow.memoization INFO: Task 949 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c292980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 949 try 0 launched on executor multi with executor id 708 parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/f7c4bf0b-186b-46ce-97f6-da7f54147c83_characterizeImage_898800_2.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/f7c4bf0b-186b-46ce-97f6-da7f54147c83_characterizeImage_898800_2.stderr parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/f5ae34b4-acc7-441a-8121-e300214535b0_isr_898800_2.stdout parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/f5ae34b4-acc7-441a-8121-e300214535b0_isr_898800_2.stderr parsl.dataflow.dflow INFO: Task 864 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 339a5000edd627e46c2d6e6631f62cdd with result from task 864 parsl.dataflow.dflow DEBUG: Task 864 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 865 has memoization hash 63d4f222da1c4d65b60b79252ba66af6 parsl.dataflow.memoization INFO: Task 865 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c441bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 865 try 0 launched on executor multi with executor id 709 parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/8b08f397-dc5f-4f99-ae24-49710aa06755_characterizeImage_898694_30.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/8b08f397-dc5f-4f99-ae24-49710aa06755_characterizeImage_898694_30.stderr parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/cb217243-9466-46a6-ad0b-680f291880d2_isr_898694_30.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/cb217243-9466-46a6-ad0b-680f291880d2_isr_898694_30.stderr parsl.dataflow.dflow INFO: Task 867 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a866cde8379499ce1d060e6226b6e9b with result from task 867 parsl.dataflow.dflow DEBUG: Task 867 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 868 has memoization hash cfee51f45ffca0a977bba5911e91f084 parsl.dataflow.memoization INFO: Task 868 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c441ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 868 try 0 launched on executor multi with executor id 710 parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/89eb401e-aa63-4301-9602-42dba77dd18e_characterizeImage_898694_11.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/89eb401e-aa63-4301-9602-42dba77dd18e_characterizeImage_898694_11.stderr parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/7e6a9095-e51f-41e1-8e85-2ee6778c8710_isr_898694_11.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/7e6a9095-e51f-41e1-8e85-2ee6778c8710_isr_898694_11.stderr parsl.dataflow.dflow INFO: Task 939 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e35c7dc1fbb6c8d5e43011778c55902 with result from task 939 parsl.dataflow.dflow DEBUG: Task 939 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 940 has memoization hash 563a0e5f330b82c63dbe453d54e75540 parsl.dataflow.memoization INFO: Task 940 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c292160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 940 try 0 launched on executor multi with executor id 711 parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ff0948ec-257e-48f5-b058-e0a54de01984_characterizeImage_898798_21.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ff0948ec-257e-48f5-b058-e0a54de01984_characterizeImage_898798_21.stderr parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/f87600ed-2b18-487f-ae2e-765a70c10a81_isr_898798_21.stdout parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/f87600ed-2b18-487f-ae2e-765a70c10a81_isr_898798_21.stderr parsl.dataflow.dflow INFO: Task 858 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d5b9f4bfeddb1aec50214ed0c7ee061 with result from task 858 parsl.dataflow.dflow DEBUG: Task 858 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 859 has memoization hash 2daa96f58b5cbdeb4890ce2a076319c3 parsl.dataflow.memoization INFO: Task 859 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c441580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 859 try 0 launched on executor multi with executor id 712 parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/6fb44520-31ca-4603-8658-7f010273562a_characterizeImage_898695_30.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/6fb44520-31ca-4603-8658-7f010273562a_characterizeImage_898695_30.stderr parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/45c3ca67-b6d6-4b45-9214-584f4b64dbee_isr_898695_30.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/45c3ca67-b6d6-4b45-9214-584f4b64dbee_isr_898695_30.stderr parsl.dataflow.dflow INFO: Task 957 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d45d63147ca03f42e51be96057330f9 with result from task 957 parsl.dataflow.dflow DEBUG: Task 957 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 958 has memoization hash 1756d9c7b108818adade0d2723e32b76 parsl.dataflow.memoization INFO: Task 958 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2936a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 958 try 0 launched on executor multi with executor id 713 parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/0c6ef654-93be-490f-9d94-d6e3faed54a7_characterizeImage_898798_51.stdout parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/0c6ef654-93be-490f-9d94-d6e3faed54a7_characterizeImage_898798_51.stderr parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/942f86b5-b8b7-423d-aeaf-7154bea804c1_isr_898798_51.stdout parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/942f86b5-b8b7-423d-aeaf-7154bea804c1_isr_898798_51.stderr parsl.dataflow.dflow INFO: Task 933 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cef45b7226cf882690f7fc055003b34f with result from task 933 parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:04 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow DEBUG: Task 933 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 934 has memoization hash 5f0520a6ae3d96d6759f77b5f190e7b4 parsl.dataflow.memoization INFO: Task 934 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c291b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 934 try 0 launched on executor multi with executor id 714 parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/0dab2b7d-1a60-4de5-afd4-334236346842_characterizeImage_898696_33.stdout parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/0dab2b7d-1a60-4de5-afd4-334236346842_characterizeImage_898696_33.stderr parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/9e7d918f-917f-40e8-8e47-be7a18be6f45_isr_898696_33.stdout parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/9e7d918f-917f-40e8-8e47-be7a18be6f45_isr_898696_33.stderr parsl.dataflow.dflow INFO: Task 981 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1061638247b5e51ad7831b6a07931544 with result from task 981 parsl.dataflow.dflow DEBUG: Task 981 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 982 has memoization hash 41d455a6bec999c126225bcea9ca351c parsl.dataflow.memoization INFO: Task 982 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 982 try 0 launched on executor multi with executor id 715 parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/ccc294d5-d3bd-499f-b185-ffc22508a410_characterizeImage_898696_34.stdout parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/ccc294d5-d3bd-499f-b185-ffc22508a410_characterizeImage_898696_34.stderr parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/c1796cee-3051-4d87-889b-7692d5d7ea9f_isr_898696_34.stdout parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/c1796cee-3051-4d87-889b-7692d5d7ea9f_isr_898696_34.stderr parsl.dataflow.dflow INFO: Task 996 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdb02983a0e0bd09c9b0a4faeb030690 with result from task 996 parsl.dataflow.dflow DEBUG: Task 996 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 997 has memoization hash 757b2310d9f2984b358e3f2423b79800 parsl.dataflow.memoization INFO: Task 997 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2da5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 997 try 0 launched on executor multi with executor id 716 parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/89ea6120-78ff-4c59-819b-33a55cc30e05_characterizeImage_898798_19.stdout parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/89ea6120-78ff-4c59-819b-33a55cc30e05_characterizeImage_898798_19.stderr parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/34e14fc7-c24b-4ab4-a7e7-30fc1824f167_isr_898798_19.stdout parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/34e14fc7-c24b-4ab4-a7e7-30fc1824f167_isr_898798_19.stderr parsl.dataflow.dflow INFO: Task 978 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 331f5206bc5623b482504a0c6fc85427 with result from task 978 parsl.dataflow.dflow DEBUG: Task 978 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 979 has memoization hash e03a77d893589d9c95913d19cfec40b3 parsl.dataflow.memoization INFO: Task 979 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 979 try 0 launched on executor multi with executor id 717 parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/81ba27f1-29c9-43f4-bd71-376068c42405_characterizeImage_898800_21.stdout parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/81ba27f1-29c9-43f4-bd71-376068c42405_characterizeImage_898800_21.stderr parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/4ace23da-a838-4ce8-a91a-309b4a57afb2_isr_898800_21.stdout parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/4ace23da-a838-4ce8-a91a-309b4a57afb2_isr_898800_21.stderr parsl.dataflow.dflow INFO: Task 963 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57fac441bc04f526214fd85b25384537 with result from task 963 parsl.dataflow.dflow DEBUG: Task 963 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 964 has memoization hash 5aec3021e41463df3c5593077b22d8e4 parsl.dataflow.memoization INFO: Task 964 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 964 try 0 launched on executor multi with executor id 718 parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/3793ed74-8e63-47a9-9c1d-655dd3a24540_characterizeImage_898799_51.stdout parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/3793ed74-8e63-47a9-9c1d-655dd3a24540_characterizeImage_898799_51.stderr parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/4ea13f91-3392-4436-8ac4-0255d857cd23_isr_898799_51.stdout parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/4ea13f91-3392-4436-8ac4-0255d857cd23_isr_898799_51.stderr parsl.dataflow.dflow INFO: Task 936 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ed308d914577fab3fc6680ad3f34a67 with result from task 936 parsl.dataflow.dflow DEBUG: Task 936 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 937 has memoization hash fdb6cf5ec2d6b82900be48dfe8f172ab parsl.dataflow.memoization INFO: Task 937 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c291e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 937 try 0 launched on executor multi with executor id 719 parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/253d9316-c469-4212-8d90-f6c1ca3c42fd_characterizeImage_898695_28.stdout parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/253d9316-c469-4212-8d90-f6c1ca3c42fd_characterizeImage_898695_28.stderr parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/80751463-1bac-4b6e-8d93-f844ac5a7b23_isr_898695_28.stdout parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/80751463-1bac-4b6e-8d93-f844ac5a7b23_isr_898695_28.stderr parsl.dataflow.dflow INFO: Task 984 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8be52af19f518a72d39b5fe27c98fe70 with result from task 984 parsl.dataflow.dflow DEBUG: Task 984 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 985 has memoization hash 6e5d37e54b81bfda4aef3069745772f6 parsl.dataflow.memoization INFO: Task 985 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 985 try 0 launched on executor multi with executor id 720 parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/d536fa26-e9ad-490a-962e-9cd0623ecf87_characterizeImage_898798_12.stdout parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/d536fa26-e9ad-490a-962e-9cd0623ecf87_characterizeImage_898798_12.stderr parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/28db2c3d-f2a4-49db-9793-9f3b7eda582b_isr_898798_12.stdout parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/28db2c3d-f2a4-49db-9793-9f3b7eda582b_isr_898798_12.stderr parsl.dataflow.dflow INFO: Task 975 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85a1d611edba099b8eee9165d0b2f417 with result from task 975 parsl.dataflow.dflow DEBUG: Task 975 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 976 has memoization hash 1b18258c1503b9bfe586b1ebcd4aa4af parsl.dataflow.memoization INFO: Task 976 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 976 try 0 launched on executor multi with executor id 721 parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/4a724df4-da41-432e-ad2d-7c5b3d81f45f_characterizeImage_898799_57.stdout parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/4a724df4-da41-432e-ad2d-7c5b3d81f45f_characterizeImage_898799_57.stderr parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/a50169d3-940b-4f30-bf83-7eac842ddd58_isr_898799_57.stdout parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/a50169d3-940b-4f30-bf83-7eac842ddd58_isr_898799_57.stderr parsl.dataflow.dflow INFO: Task 990 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 288a89d2c2129d6baf1063e9b14df990 with result from task 990 parsl.dataflow.dflow DEBUG: Task 990 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 991 has memoization hash 6c2a59281824f8a0c10f9afa7a9bd34d parsl.dataflow.memoization INFO: Task 991 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 991 try 0 launched on executor multi with executor id 722 parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/bd18feba-85c1-4636-974c-48971fbe0f4c_characterizeImage_898694_61.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/bd18feba-85c1-4636-974c-48971fbe0f4c_characterizeImage_898694_61.stderr parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/09a3c603-27d1-4ae5-84d2-ecf4610c67e5_isr_898694_61.stdout parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/09a3c603-27d1-4ae5-84d2-ecf4610c67e5_isr_898694_61.stderr parsl.dataflow.dflow INFO: Task 999 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a370f0b0b1a480926948e92b00298ea with result from task 999 parsl.dataflow.dflow DEBUG: Task 999 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1000 has memoization hash 369a19cd5ce86588e60a2ed3bb5d6db5 parsl.dataflow.memoization INFO: Task 1000 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2da8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1000 try 0 launched on executor multi with executor id 723 parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/e1da4da7-ed11-492a-9c04-e1d2a35b411a_characterizeImage_898695_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/e1da4da7-ed11-492a-9c04-e1d2a35b411a_characterizeImage_898695_42.stderr parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/b05f1916-e33d-4ce4-ab19-02f5cfcfafcf_isr_898695_42.stdout parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/b05f1916-e33d-4ce4-ab19-02f5cfcfafcf_isr_898695_42.stderr parsl.dataflow.dflow INFO: Task 954 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69dd30147106005772c929630b1f8556 with result from task 954 parsl.dataflow.dflow DEBUG: Task 954 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 955 has memoization hash 5459f04ef570b53e332c5262d9bf60a7 parsl.dataflow.memoization INFO: Task 955 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c293100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 955 try 0 launched on executor multi with executor id 724 parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ef6314c9-413b-4baa-8757-d43b37701062_characterizeImage_898798_1.stdout parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ef6314c9-413b-4baa-8757-d43b37701062_characterizeImage_898798_1.stderr parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/24352de4-a006-4174-a5b2-f0c6ae7d5323_isr_898798_1.stdout parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/24352de4-a006-4174-a5b2-f0c6ae7d5323_isr_898798_1.stderr parsl.dataflow.dflow INFO: Task 870 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e9d4747ccad2e86b387c758bb67b0bf with result from task 870 parsl.dataflow.dflow DEBUG: Task 870 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 871 has memoization hash c36aed07051d255aa6581eff73475106 parsl.dataflow.memoization INFO: Task 871 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c442200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 871 try 0 launched on executor multi with executor id 725 parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/95fd5844-b9eb-40d2-b532-9daf59eab2b8_characterizeImage_898696_54.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/95fd5844-b9eb-40d2-b532-9daf59eab2b8_characterizeImage_898696_54.stderr parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/f1c4813a-5180-4788-a641-a5180d1fa9bd_isr_898696_54.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/f1c4813a-5180-4788-a641-a5180d1fa9bd_isr_898696_54.stderr parsl.dataflow.dflow INFO: Task 873 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45df10f7cdb032381ba6a3e8770d9a57 with result from task 873 parsl.dataflow.dflow DEBUG: Task 873 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 874 has memoization hash ee8993e920e0bb103fc95bb9b60c287b parsl.dataflow.memoization INFO: Task 874 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c442520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 874 try 0 launched on executor multi with executor id 726 parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/c8858d83-ca4a-4ea3-875c-de04eccf789c_characterizeImage_898694_37.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/c8858d83-ca4a-4ea3-875c-de04eccf789c_characterizeImage_898694_37.stderr parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/6bcc0346-1294-4936-ae91-9243f273f472_isr_898694_37.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/6bcc0346-1294-4936-ae91-9243f273f472_isr_898694_37.stderr parsl.dataflow.dflow INFO: Task 987 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dfcd2138dbfa0b654eb5e09737d82af with result from task 987 parsl.dataflow.dflow DEBUG: Task 987 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 988 has memoization hash 722fcf022c936e3935898a266ad80fd7 parsl.dataflow.memoization INFO: Task 988 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 988 try 0 launched on executor multi with executor id 727 parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/1ffb46b2-7a14-495a-9a07-5d8c26b6ad76_characterizeImage_898695_61.stdout parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/1ffb46b2-7a14-495a-9a07-5d8c26b6ad76_characterizeImage_898695_61.stderr parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/5a923c84-5011-4e29-b3a7-54626f2345ae_isr_898695_61.stdout parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/5a923c84-5011-4e29-b3a7-54626f2345ae_isr_898695_61.stderr parsl.dataflow.dflow INFO: Task 861 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6ef58002c2537e20851802f4ab2fc5a with result from task 861 parsl.dataflow.dflow DEBUG: Task 861 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 862 has memoization hash 2faa22978d559de173ea698c590413ba parsl.dataflow.memoization INFO: Task 862 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4418a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 862 try 0 launched on executor multi with executor id 728 parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/d9fc4a9c-cdcf-46c3-96d9-bfcdff34eb4f_characterizeImage_898696_7.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/d9fc4a9c-cdcf-46c3-96d9-bfcdff34eb4f_characterizeImage_898696_7.stderr parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/8a51d2ab-c486-4aa8-b964-432e3ba688d0_isr_898696_7.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/8a51d2ab-c486-4aa8-b964-432e3ba688d0_isr_898696_7.stderr parsl.dataflow.dflow INFO: Task 945 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 298f60f989a0cd803bf0dfbbf3a87b19 with result from task 945 parsl.dataflow.dflow DEBUG: Task 945 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 946 has memoization hash 5bac2038a4344c51455696d63aa7c94d parsl.dataflow.memoization INFO: Task 946 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c292660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 946 try 0 launched on executor multi with executor id 729 parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/a61146b2-edf4-4c83-b83c-f41607c7fd51_characterizeImage_898799_9.stdout parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/a61146b2-edf4-4c83-b83c-f41607c7fd51_characterizeImage_898799_9.stderr parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/1f43662c-6043-4ed6-b3b9-b58556c5c804_isr_898799_9.stdout parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/1f43662c-6043-4ed6-b3b9-b58556c5c804_isr_898799_9.stderr parsl.dataflow.dflow INFO: Task 993 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f35b9a565c0b6c36f803a1d309e6a55f with result from task 993 parsl.dataflow.dflow DEBUG: Task 993 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 994 has memoization hash b60ed8b9a2905e3b06528c740b8e67ae parsl.dataflow.memoization INFO: Task 994 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2da2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 994 try 0 launched on executor multi with executor id 730 parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ba336da1-e362-4571-a751-659e82b0d625_characterizeImage_898798_7.stdout parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ba336da1-e362-4571-a751-659e82b0d625_characterizeImage_898798_7.stderr parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/bcd85c87-3023-428f-accd-32fc43f39a67_isr_898798_7.stdout parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/bcd85c87-3023-428f-accd-32fc43f39a67_isr_898798_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:05 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 942 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f63c7a0e33c34f2564691649cbee7e7a with result from task 942 parsl.dataflow.dflow DEBUG: Task 942 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 943 has memoization hash 8ca89422d022310a130a59bef185ce80 parsl.dataflow.memoization INFO: Task 943 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c292340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 943 try 0 launched on executor multi with executor id 731 parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/c9844dfb-2978-4e5e-9da9-d236e1419f51_characterizeImage_898800_61.stdout parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/c9844dfb-2978-4e5e-9da9-d236e1419f51_characterizeImage_898800_61.stderr parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/c12239ed-1b55-421a-b23b-3e2d4aa3fdb8_isr_898800_61.stdout parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/c12239ed-1b55-421a-b23b-3e2d4aa3fdb8_isr_898800_61.stderr parsl.dataflow.dflow INFO: Task 960 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9036686abcf3d14dc46ae9ebc1fa5b9 with result from task 960 parsl.dataflow.dflow DEBUG: Task 960 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 961 has memoization hash 74e0e85d087cef490e722168c7f21432 parsl.dataflow.memoization INFO: Task 961 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2934c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 961 try 0 launched on executor multi with executor id 732 parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/1cb94f31-9003-4445-a4d9-c924b45d3f5c_characterizeImage_898800_3.stdout parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/1cb94f31-9003-4445-a4d9-c924b45d3f5c_characterizeImage_898800_3.stderr parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/528f690a-7c33-479c-a3a9-8567587011eb_isr_898800_3.stdout parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/528f690a-7c33-479c-a3a9-8567587011eb_isr_898800_3.stderr parsl.dataflow.dflow INFO: Task 930 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed71ef4d5295ef94c1fcd04721b12c99 with result from task 930 parsl.dataflow.dflow DEBUG: Task 930 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 931 has memoization hash 41acb0d6322d657a06bf4053830e6615 parsl.dataflow.memoization INFO: Task 931 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c291580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 931 try 0 launched on executor multi with executor id 733 parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/fc370d9a-1c80-4e82-9479-05c4e817ee26_characterizeImage_898694_9.stdout parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/fc370d9a-1c80-4e82-9479-05c4e817ee26_characterizeImage_898694_9.stderr parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/415cbfb9-bfd1-4903-a24e-71da1df10682_isr_898694_9.stdout parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898694/415cbfb9-bfd1-4903-a24e-71da1df10682_isr_898694_9.stderr parsl.dataflow.dflow INFO: Task 972 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b275d64ec2b91b6a13b93c23326f1ec0 with result from task 972 parsl.dataflow.dflow DEBUG: Task 972 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 973 has memoization hash 63c58605f12a45ecd8b0600be0904240 parsl.dataflow.memoization INFO: Task 973 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 973 try 0 launched on executor multi with executor id 734 parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/71d942ea-2198-413c-90c8-62e4834076a7_characterizeImage_898696_15.stdout parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/71d942ea-2198-413c-90c8-62e4834076a7_characterizeImage_898696_15.stderr parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/589ace7e-a037-4dcb-9c8f-aee3432de889_isr_898696_15.stdout parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/589ace7e-a037-4dcb-9c8f-aee3432de889_isr_898696_15.stderr parsl.dataflow.dflow INFO: Task 966 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0708c81c9e0d7a6a74c2d5804a259f22 with result from task 966 parsl.dataflow.dflow DEBUG: Task 966 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 967 has memoization hash 5df67e51a44b92b1b2f1a7c1be7c6c52 parsl.dataflow.memoization INFO: Task 967 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 967 try 0 launched on executor multi with executor id 735 parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/9e38a3e2-7ec2-4f69-8109-c563ab7bced0_characterizeImage_898695_36.stdout parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/9e38a3e2-7ec2-4f69-8109-c563ab7bced0_characterizeImage_898695_36.stderr parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/a116b3e4-de6d-4fc9-87aa-0fad11b6987b_isr_898695_36.stdout parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/a116b3e4-de6d-4fc9-87aa-0fad11b6987b_isr_898695_36.stderr parsl.dataflow.dflow INFO: Task 969 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e05afe3500ad3e46b6a4b9a00e8e45d9 with result from task 969 parsl.dataflow.dflow DEBUG: Task 969 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 970 has memoization hash f9f5ddc432108a5001d3a1c1f8aacdcc parsl.dataflow.memoization INFO: Task 970 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 970 try 0 launched on executor multi with executor id 736 parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/38f14a6f-d695-4a47-bb49-fc6ceb937657_characterizeImage_898695_20.stdout parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/38f14a6f-d695-4a47-bb49-fc6ceb937657_characterizeImage_898695_20.stderr parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/cb0ea134-aafd-498d-8f49-eb21e2717a50_isr_898695_20.stdout parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/cb0ea134-aafd-498d-8f49-eb21e2717a50_isr_898695_20.stderr parsl.dataflow.dflow INFO: Task 951 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba85e32263ccd133272544a489043059 with result from task 951 parsl.dataflow.dflow DEBUG: Task 951 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 952 has memoization hash 1629ae4109e440dcd6c4a10290ed726a parsl.dataflow.memoization INFO: Task 952 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c292de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 952 try 0 launched on executor multi with executor id 737 parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/96d9b086-7575-4b07-a154-a6c20680a85b_characterizeImage_898798_16.stdout parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/96d9b086-7575-4b07-a154-a6c20680a85b_characterizeImage_898798_16.stderr parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/89a19fed-e26a-4ef9-b449-03cd45c40ed8_isr_898798_16.stdout parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898798/89a19fed-e26a-4ef9-b449-03cd45c40ed8_isr_898798_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:06 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:08 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:09 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 918 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efa7adff388e866ead39b6432ee70edf with result from task 918 parsl.dataflow.dflow DEBUG: Task 918 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 919 has memoization hash 82812a34f76104b2922b44e87a037a5e parsl.dataflow.memoization INFO: Task 919 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c291260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 919 try 0 launched on executor multi with executor id 738 parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/edf29467-f313-40b0-b03d-667ff3933dfd_characterizeImage_898800_5.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/edf29467-f313-40b0-b03d-667ff3933dfd_characterizeImage_898800_5.stderr parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/e2814dbe-7be7-49e0-8382-3d36c39f8f57_isr_898800_5.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/e2814dbe-7be7-49e0-8382-3d36c39f8f57_isr_898800_5.stderr parsl.dataflow.dflow INFO: Task 909 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd1b91e84a936d7533a3671a222e38ce with result from task 909 parsl.dataflow.dflow DEBUG: Task 909 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 910 has memoization hash 3ea21cd80cc74b6680a5069b34058473 parsl.dataflow.memoization INFO: Task 910 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c290900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 910 try 0 launched on executor multi with executor id 739 parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/6e76686c-6764-440c-a9d6-f0fb71d2ec63_characterizeImage_898800_4.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/6e76686c-6764-440c-a9d6-f0fb71d2ec63_characterizeImage_898800_4.stderr parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/b5721661-4258-4291-b5a6-cd90379789ca_isr_898800_4.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/b5721661-4258-4291-b5a6-cd90379789ca_isr_898800_4.stderr parsl.dataflow.dflow INFO: Task 906 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db57f933b01b5124e6c3369b9d50b544 with result from task 906 parsl.dataflow.dflow DEBUG: Task 906 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 907 has memoization hash a9beefb19752e7dfd64d50e8c5635c82 parsl.dataflow.memoization INFO: Task 907 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2905e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 907 try 0 launched on executor multi with executor id 740 parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/44dc3ea1-a77a-49d3-ac50-ac69c2b47ab6_characterizeImage_898696_3.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/44dc3ea1-a77a-49d3-ac50-ac69c2b47ab6_characterizeImage_898696_3.stderr parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/d15b739b-9b44-4c52-a722-1a77642503b8_isr_898696_3.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/d15b739b-9b44-4c52-a722-1a77642503b8_isr_898696_3.stderr parsl.dataflow.dflow INFO: Task 915 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07ae30a681c21c17e9cac492677ae7a6 with result from task 915 parsl.dataflow.dflow DEBUG: Task 915 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 916 has memoization hash f33c95d8dc2eea07f6e0df1da6f1c1cd parsl.dataflow.memoization INFO: Task 916 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c290f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 916 try 0 launched on executor multi with executor id 741 parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/03ac09a1-68db-4bc4-94f7-acd4f99cc6c5_characterizeImage_898800_14.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/03ac09a1-68db-4bc4-94f7-acd4f99cc6c5_characterizeImage_898800_14.stderr parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/27dde754-135c-4b11-b245-0385e7122dfa_isr_898800_14.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898800/27dde754-135c-4b11-b245-0385e7122dfa_isr_898800_14.stderr parsl.dataflow.dflow INFO: Task 927 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 642b1d07959ed926cfaa1f4ce8195965 with result from task 927 parsl.dataflow.dflow DEBUG: Task 927 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 928 has memoization hash 5b0fddba10327a7b49f7235abfd0646f parsl.dataflow.memoization INFO: Task 928 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c293600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 928 try 0 launched on executor multi with executor id 742 parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/2e111535-a3d1-484f-8fe1-155b7def2e12_characterizeImage_898696_16.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/2e111535-a3d1-484f-8fe1-155b7def2e12_characterizeImage_898696_16.stderr parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/3056ed80-d157-44a2-9e32-6edd6be6f185_isr_898696_16.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898696/3056ed80-d157-44a2-9e32-6edd6be6f185_isr_898696_16.stderr parsl.dataflow.dflow INFO: Task 912 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ce011445cfa4569b2e0dd62cbf15874 with result from task 912 parsl.dataflow.dflow DEBUG: Task 912 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 913 has memoization hash ea2fdfe774ca1a74d24125cd2cf1a2a7 parsl.dataflow.memoization INFO: Task 913 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c290c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 913 try 0 launched on executor multi with executor id 743 parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/c4972256-4b87-4038-94b3-dbe1eb4c41c4_characterizeImage_898799_12.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/c4972256-4b87-4038-94b3-dbe1eb4c41c4_characterizeImage_898799_12.stderr parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/01802909-7f15-4e3d-99d4-3d284d3c149c_isr_898799_12.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898799/01802909-7f15-4e3d-99d4-3d284d3c149c_isr_898799_12.stderr parsl.dataflow.dflow INFO: Task 921 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e0844a347f35e64fdd56ed5792b524b with result from task 921 parsl.dataflow.dflow DEBUG: Task 921 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 922 has memoization hash a79a82c27da9bf8cec106e76321500ab parsl.dataflow.memoization INFO: Task 922 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c293c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 922 try 0 launched on executor multi with executor id 744 parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/58d9de2b-fffd-4eab-b61f-287c21bbadf7_characterizeImage_898695_13.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/58d9de2b-fffd-4eab-b61f-287c21bbadf7_characterizeImage_898695_13.stderr parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/13930b43-43a3-42be-bbe5-f1010b32489e_isr_898695_13.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/13930b43-43a3-42be-bbe5-f1010b32489e_isr_898695_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:10 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 924 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a7ea0884b7fb0552b9e537c0da293bb with result from task 924 parsl.dataflow.dflow DEBUG: Task 924 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 925 has memoization hash 59e225bcfd2056f6b822cda7c4248559 parsl.dataflow.memoization INFO: Task 925 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c293920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 925 try 0 launched on executor multi with executor id 745 parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/70f44ad4-c1ac-460c-bbfe-efd4868d8f0a_characterizeImage_898695_50.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/70f44ad4-c1ac-460c-bbfe-efd4868d8f0a_characterizeImage_898695_50.stderr parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/49a1d469-b30e-46f9-a168-b862ce51b6c3_isr_898695_50.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/isr/898695/49a1d469-b30e-46f9-a168-b862ce51b6c3_isr_898695_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:11 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:13 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:14 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:15 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:17 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:18 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:19 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 949 try 0 failed parsl.dataflow.dflow INFO: Task 949 marked for retry parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/f7c4bf0b-186b-46ce-97f6-da7f54147c83_characterizeImage_898800_2.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/f7c4bf0b-186b-46ce-97f6-da7f54147c83_characterizeImage_898800_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 949 has memoization hash 17940b7b86211d58e6922cf4d68cf1ff parsl.dataflow.memoization INFO: Task 949 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c292980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 949 try 1 launched on executor multi with executor id 746 parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/f7c4bf0b-186b-46ce-97f6-da7f54147c83_characterizeImage_898800_2.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/f7c4bf0b-186b-46ce-97f6-da7f54147c83_characterizeImage_898800_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:21 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:22 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:23 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:25 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:26 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 457 try 0 failed parsl.dataflow.dflow INFO: Task 457 marked for retry parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/b06bb2c5-1ca2-44e3-ba6d-40832489814f_characterizeImage_898798_2.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/b06bb2c5-1ca2-44e3-ba6d-40832489814f_characterizeImage_898798_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 457 has memoization hash 1378e62d0a8735467f133c64ddf613fc parsl.dataflow.memoization INFO: Task 457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 457 try 1 launched on executor multi with executor id 747 parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/b06bb2c5-1ca2-44e3-ba6d-40832489814f_characterizeImage_898798_2.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/b06bb2c5-1ca2-44e3-ba6d-40832489814f_characterizeImage_898798_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:27 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:29 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:30 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 352 try 0 failed parsl.dataflow.dflow INFO: Task 352 marked for retry parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/645d169e-9684-4549-8e89-89431d6f7a50_characterizeImage_898696_2.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/645d169e-9684-4549-8e89-89431d6f7a50_characterizeImage_898696_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 77d7575d667b9ad34c090c8f1676287a parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 1 launched on executor multi with executor id 748 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/645d169e-9684-4549-8e89-89431d6f7a50_characterizeImage_898696_2.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/645d169e-9684-4549-8e89-89431d6f7a50_characterizeImage_898696_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:32 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 244 try 1 failed parsl.dataflow.dflow ERROR: Task 244 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry aaefa86c8314ae4c1be5230d360abe77 with result from task 244 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 245 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 245 try 0 failed parsl.dataflow.dflow INFO: Task 245 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/c5c69d36-cece-4fe1-aa83-a18aa4700fb5_calibrate_898799_2.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/c5c69d36-cece-4fe1-aa83-a18aa4700fb5_calibrate_898799_2.stderr parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/a91461ad-3730-4ccd-81a7-8a82038a38a0_characterizeImage_898799_2.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/a91461ad-3730-4ccd-81a7-8a82038a38a0_characterizeImage_898799_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:33 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:35 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:36 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffd643eefe09c50fb5a8d367d5d8a713 with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 753817b9665d0036831e69cca439d5a9 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 749 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/0061690f-b2dc-4c25-82ed-1df158ed7e06_calibrate_898800_31.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/0061690f-b2dc-4c25-82ed-1df158ed7e06_calibrate_898800_31.stderr parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/bb5eb107-4898-4b6d-bb02-83aa669e3d94_characterizeImage_898800_31.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/bb5eb107-4898-4b6d-bb02-83aa669e3d94_characterizeImage_898800_31.stderr parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4d6ce57811994fe9a456bb11653fdbd with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash ed1f0b65c5a699a3f496da5e186f8b4c parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5eda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 750 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/1518567d-1ff8-49c7-bee9-3513c642630d_calibrate_898694_31.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/1518567d-1ff8-49c7-bee9-3513c642630d_calibrate_898694_31.stderr parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/ae0e9aa1-5eab-4184-a6fe-35869c139b00_characterizeImage_898694_31.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/ae0e9aa1-5eab-4184-a6fe-35869c139b00_characterizeImage_898694_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:38 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:39 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:40 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:42 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:43 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:44 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51aeac3e81722f53471e17a86327dcdb with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 4ffd425ff914452438e66c03e0f58a1b parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5868e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 751 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/fd362619-6014-4122-8a6f-62f5916bb2f4_calibrate_898696_42.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/fd362619-6014-4122-8a6f-62f5916bb2f4_calibrate_898696_42.stderr parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/9449de46-25aa-4784-8a85-07e8a6950582_characterizeImage_898696_42.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/9449de46-25aa-4784-8a85-07e8a6950582_characterizeImage_898696_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:46 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:47 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d91726bd6eee250267604a65d7011cdd with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 3203c0344fa1da7673d9866178eff350 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ccf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 752 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/6cff600c-4bfd-4346-9f8d-48b61977e1f0_calibrate_898694_53.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/6cff600c-4bfd-4346-9f8d-48b61977e1f0_calibrate_898694_53.stderr parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/59abbccc-be23-428b-a971-1631d98a3087_characterizeImage_898694_53.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/59abbccc-be23-428b-a971-1631d98a3087_characterizeImage_898694_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:48 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16070d339a581124a5b4a6588ac9dc5f with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash 725019a58421836aaa5b7f22d2cd44ae parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d608220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 753 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/b913b389-7f8a-4427-8f77-a275b23101cd_calibrate_898696_21.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/b913b389-7f8a-4427-8f77-a275b23101cd_calibrate_898696_21.stderr parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/65b2b50e-6819-4b89-8fea-413b4184befe_characterizeImage_898696_21.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/65b2b50e-6819-4b89-8fea-413b4184befe_characterizeImage_898696_21.stderr parsl.dataflow.dflow INFO: Task 682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c2d9345a6cae992e283a125fc0c3108 with result from task 682 parsl.dataflow.dflow DEBUG: Task 682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 683 has memoization hash 6c4bc9520bf83b6fbb2921d06d72fa8d parsl.dataflow.memoization INFO: Task 683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3751c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 683 try 0 launched on executor multi with executor id 754 parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/754d2c16-e782-4c12-abeb-8ec38cab3f1c_calibrate_898799_31.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/754d2c16-e782-4c12-abeb-8ec38cab3f1c_calibrate_898799_31.stderr parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/72d3e4dd-f3ea-4611-9daf-34e9c71ce583_characterizeImage_898799_31.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/72d3e4dd-f3ea-4611-9daf-34e9c71ce583_characterizeImage_898799_31.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 017d4eae42dadb22eae70b6d213a98d7 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 564e2a67468e0bb462a2980c9144010f parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cd580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 755 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/934390cc-cb46-435a-bfb6-c210af01e674_calibrate_898694_57.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/934390cc-cb46-435a-bfb6-c210af01e674_calibrate_898694_57.stderr parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/c48e7e81-a51f-4e96-b8cc-9e748f0f4022_characterizeImage_898694_57.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/c48e7e81-a51f-4e96-b8cc-9e748f0f4022_characterizeImage_898694_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:50 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54a5b617db774d3163e9dbd0040d6f9d with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash a3c0b1a4e00a2ac3df3b1bcad1560679 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d587880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 756 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/735355a5-f5c5-49f1-907c-798f99d54351_calibrate_898695_62.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/735355a5-f5c5-49f1-907c-798f99d54351_calibrate_898695_62.stderr parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/746d5ff3-0e87-4b04-840a-e77c768bdf3e_characterizeImage_898695_62.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/746d5ff3-0e87-4b04-840a-e77c768bdf3e_characterizeImage_898695_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:51 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b83682532f2d74783c4b9c6f49e8bc33 with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash 23f85b21ec57e21da129e8b3390d66e8 parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 757 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/edb4db20-eb85-4ddc-ac72-0e6ec9f4ae51_calibrate_898694_10.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/edb4db20-eb85-4ddc-ac72-0e6ec9f4ae51_calibrate_898694_10.stderr parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/706e310d-5e2a-4a1b-abc3-c6f874be370c_characterizeImage_898694_10.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/706e310d-5e2a-4a1b-abc3-c6f874be370c_characterizeImage_898694_10.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82cfc3c5210f881400db597d5a17e203 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash c12e160036598ff12fecd87aebd94d06 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cdee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 758 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/58ce860f-940c-4f3d-9bf3-c17986ea3191_calibrate_898800_40.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/58ce860f-940c-4f3d-9bf3-c17986ea3191_calibrate_898800_40.stderr parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/4f269a1b-e601-47d7-b2fe-1926d46d4d37_characterizeImage_898800_40.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/4f269a1b-e601-47d7-b2fe-1926d46d4d37_characterizeImage_898800_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:53 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:54 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4daac0cac00a2083b368424bc78472d with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash 7b63cc58e176875bf90e977f54383af4 parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d609300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 759 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/d1b50ea5-55b9-44dc-99e7-1d53d0008530_calibrate_898694_42.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/d1b50ea5-55b9-44dc-99e7-1d53d0008530_calibrate_898694_42.stderr parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/9bdaa6db-cbb2-4d21-98b0-d5e25045cdfb_characterizeImage_898694_42.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/9bdaa6db-cbb2-4d21-98b0-d5e25045cdfb_characterizeImage_898694_42.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7356d73e7f6d1d85d93510ebf88b886c with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 5515f18c2e3d31d45e2e92243f7915f8 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5aa980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 760 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/01365429-6f46-4101-833a-4c7ca79eaf7f_calibrate_898695_47.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/01365429-6f46-4101-833a-4c7ca79eaf7f_calibrate_898695_47.stderr parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/409fdf89-7480-4668-96af-7cb33b6dd307_characterizeImage_898695_47.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/409fdf89-7480-4668-96af-7cb33b6dd307_characterizeImage_898695_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:55 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:56 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 902f28757ee042fd668c467bd2819eb1 with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 78de77abb2d3f2dec12829bff1cc0e1d parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ce200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 761 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/d855c6bd-e0c6-4f51-83ed-0fd911525ca7_calibrate_898799_10.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/d855c6bd-e0c6-4f51-83ed-0fd911525ca7_calibrate_898799_10.stderr parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/ad5c3957-f1d5-49e9-aed7-fbd35ce9bebf_characterizeImage_898799_10.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/ad5c3957-f1d5-49e9-aed7-fbd35ce9bebf_characterizeImage_898799_10.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a3a1e1adfb881b080886d3d130beeb7 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 17308b273f4098280f5f430aac73ec7a parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cc5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 762 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/8a153b53-76b8-46e1-9177-66bc09a2412d_calibrate_898800_54.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/8a153b53-76b8-46e1-9177-66bc09a2412d_calibrate_898800_54.stderr parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/852c74f9-255e-47a6-b613-ce709d7432b6_characterizeImage_898800_54.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/852c74f9-255e-47a6-b613-ce709d7432b6_characterizeImage_898800_54.stderr parsl.dataflow.dflow INFO: Task 709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e9b355ac0df3c0df951e5e36c1824de with result from task 709 parsl.dataflow.dflow DEBUG: Task 709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 710 has memoization hash ceee281bca13545cc08e24636953e0ed parsl.dataflow.memoization INFO: Task 710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3776a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 710 try 0 launched on executor multi with executor id 763 parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/b812b617-2bce-4db0-b4cf-ca4cb1beeb5d_calibrate_898696_57.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/b812b617-2bce-4db0-b4cf-ca4cb1beeb5d_calibrate_898696_57.stderr parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/68aeb1ba-3cd6-494d-88ca-919882b0dc15_characterizeImage_898696_57.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/68aeb1ba-3cd6-494d-88ca-919882b0dc15_characterizeImage_898696_57.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea9caa3428e33940898d1e18f7f3783b with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 5092cbbb3cf79db9fe82400679103b74 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d585e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 764 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/8be4a658-ffbe-4bc8-a401-13cf130230b1_calibrate_898798_57.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/8be4a658-ffbe-4bc8-a401-13cf130230b1_calibrate_898798_57.stderr parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/d14e9a83-2465-41cb-b854-10f0f2a645c6_characterizeImage_898798_57.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/d14e9a83-2465-41cb-b854-10f0f2a645c6_characterizeImage_898798_57.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22718c97fb8ebc7d147ae91fc22a202e with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 8133e8839f59a425f20c475716677e51 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cc4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 765 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/f2812508-890d-423c-ae36-62e33959fe45_calibrate_898694_62.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/f2812508-890d-423c-ae36-62e33959fe45_calibrate_898694_62.stderr parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/b414abf4-c8f6-45cc-9c49-9a8c3c244201_characterizeImage_898694_62.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/b414abf4-c8f6-45cc-9c49-9a8c3c244201_characterizeImage_898694_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:58 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40807a35a5c5def4c8ba29d34bc9d031 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash b78463ea5f552d541641ea1f5f391932 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5851c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 766 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/9ca2c801-ddfe-419c-94b8-1dfdcc7e4a9b_calibrate_898696_36.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/9ca2c801-ddfe-419c-94b8-1dfdcc7e4a9b_calibrate_898696_36.stderr parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/4783d71a-270a-4a6a-b9b8-6f8bad961d4c_characterizeImage_898696_36.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/4783d71a-270a-4a6a-b9b8-6f8bad961d4c_characterizeImage_898696_36.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a39b3fdae4c02622aad7758dbae4f055 with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 540edd5fbf28d4c0eb75c315e83e94df parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cdbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 767 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/51da929e-70e7-4d64-ac8e-8919967fa8fa_calibrate_898694_47.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/51da929e-70e7-4d64-ac8e-8919967fa8fa_calibrate_898694_47.stderr parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/5f432787-f449-4c8d-82e5-8fe91f9368dd_characterizeImage_898694_47.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/5f432787-f449-4c8d-82e5-8fe91f9368dd_characterizeImage_898694_47.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f45af7a8a0811341607a540f24679807 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 823f9fac37660abe9e908ec46c47bdec parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5867a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 768 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/8a1ab9fa-3f1b-4a5d-ade1-1f5024240b20_calibrate_898694_18.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/8a1ab9fa-3f1b-4a5d-ade1-1f5024240b20_calibrate_898694_18.stderr parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/9df8d6f0-6829-4717-b3c1-b3ab1a72b673_characterizeImage_898694_18.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/9df8d6f0-6829-4717-b3c1-b3ab1a72b673_characterizeImage_898694_18.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c3fce9c8311edd1083b2d151006d1f7 with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 05e4b5b5c4cb21cce26d8d7452dba4f0 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 769 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/93888075-38d6-4169-b6a6-255d28f9c246_calibrate_898800_7.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/93888075-38d6-4169-b6a6-255d28f9c246_calibrate_898800_7.stderr parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/aba1cf11-06c3-4ef4-8432-6eee53abd353_characterizeImage_898800_7.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/aba1cf11-06c3-4ef4-8432-6eee53abd353_characterizeImage_898800_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:47:59 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:00 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17f33406232056a6f78ec2f524ecded5 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 14b8c398cd694627da09431175408f8b parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d584fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 770 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/25e2fc2e-6cc4-4683-a3be-0168dd7242f3_calibrate_898696_1.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/25e2fc2e-6cc4-4683-a3be-0168dd7242f3_calibrate_898696_1.stderr parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/7b06b4e9-8f2b-4d2e-9547-3158c6d7d943_characterizeImage_898696_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/7b06b4e9-8f2b-4d2e-9547-3158c6d7d943_characterizeImage_898696_1.stderr parsl.dataflow.dflow INFO: Task 469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfad0a91d41de9b78146f8fb5c51dc72 with result from task 469 parsl.dataflow.dflow DEBUG: Task 469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 470 has memoization hash 7cea5a5b59c4f9869387fe0a6e24d37d parsl.dataflow.memoization INFO: Task 470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4823e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 771 parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/1bf5a92a-8c33-43bc-960a-ed52abb5655a_calibrate_898798_31.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/1bf5a92a-8c33-43bc-960a-ed52abb5655a_calibrate_898798_31.stderr parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/51813a66-c61b-41af-8bf3-5f69d5bf333a_characterizeImage_898798_31.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/51813a66-c61b-41af-8bf3-5f69d5bf333a_characterizeImage_898798_31.stderr parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b3ee609a840e6de67f57c4027c793d2 with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 70592edfdf3163ec66db34c699071322 parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 772 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/3586e44c-9e67-463b-acfe-b66dd66ce980_calibrate_898799_47.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/3586e44c-9e67-463b-acfe-b66dd66ce980_calibrate_898799_47.stderr parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/36fbb940-b7a5-4d69-ac6f-5f9c08c5ef00_characterizeImage_898799_47.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/36fbb940-b7a5-4d69-ac6f-5f9c08c5ef00_characterizeImage_898799_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:02 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8195b1a3fee85fc51d30e88d88ced5e2 with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash f4262f38f713e978ae00da2abf3818ee parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5efce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 773 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/45df528a-d5f9-449a-9f65-789721b06b56_calibrate_898798_41.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/45df528a-d5f9-449a-9f65-789721b06b56_calibrate_898798_41.stderr parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/2567e18b-c380-43b5-85d4-b32777321532_characterizeImage_898798_41.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/2567e18b-c380-43b5-85d4-b32777321532_characterizeImage_898798_41.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f7291110b9f8d87bfb9ddb369345223 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 2c20de635456102991cfd3c8542a7982 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5aaca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 774 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/4ecf9520-6035-41cd-834d-fd4980285f17_calibrate_898694_48.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/4ecf9520-6035-41cd-834d-fd4980285f17_calibrate_898694_48.stderr parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/c6ce5c18-d2d6-4ac6-b466-5aa3149e2094_characterizeImage_898694_48.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/c6ce5c18-d2d6-4ac6-b466-5aa3149e2094_characterizeImage_898694_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:03 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64f7785d3720f7d0c2c5a1ad9082697d with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash e033c1304f2bf9d3ed13dbfb23d5077c parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 775 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/1ba59178-de10-4657-a371-7468053437e9_calibrate_898694_50.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/1ba59178-de10-4657-a371-7468053437e9_calibrate_898694_50.stderr parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/a06ccafd-1c79-473b-99f4-918df2279237_characterizeImage_898694_50.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/a06ccafd-1c79-473b-99f4-918df2279237_characterizeImage_898694_50.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 949 try 1 failed parsl.dataflow.dflow ERROR: Task 949 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 17940b7b86211d58e6922cf4d68cf1ff with result from task 949 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 950 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 950 try 0 failed parsl.dataflow.dflow INFO: Task 950 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/039254b6-3887-4421-a59e-2eff112a2b75_calibrate_898800_2.stdout parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/039254b6-3887-4421-a59e-2eff112a2b75_calibrate_898800_2.stderr parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/f7c4bf0b-186b-46ce-97f6-da7f54147c83_characterizeImage_898800_2.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/f7c4bf0b-186b-46ce-97f6-da7f54147c83_characterizeImage_898800_2.stderr parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fe9a56f9fa23c1feb9a606c689356b8 with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 70ade959be19df2ce339d2eebddb4d41 parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 776 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/39c85a28-5a6f-423d-96d2-e8b6156755ca_calibrate_898800_57.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/39c85a28-5a6f-423d-96d2-e8b6156755ca_calibrate_898800_57.stderr parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/d1956f79-6612-47e6-8b35-a11a603f780b_characterizeImage_898800_57.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/d1956f79-6612-47e6-8b35-a11a603f780b_characterizeImage_898800_57.stderr parsl.dataflow.dflow INFO: Task 805 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b266dcd65d9755ae9d2326872d7b26b with result from task 805 parsl.dataflow.dflow DEBUG: Task 805 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 806 has memoization hash 9deae41eaec089ec10d34d8e7d83474e parsl.dataflow.memoization INFO: Task 806 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 806 try 0 launched on executor multi with executor id 777 parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/f3536772-7870-4576-9c98-6f56a2d7af69_calibrate_898799_54.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/f3536772-7870-4576-9c98-6f56a2d7af69_calibrate_898799_54.stderr parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/5c9daef3-a5cb-4d5f-bed3-70fd2d7569cb_characterizeImage_898799_54.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/5c9daef3-a5cb-4d5f-bed3-70fd2d7569cb_characterizeImage_898799_54.stderr parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f63645b4785dca99abc7b440a9e15059 with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash f977d08a926edb85ecaf0c5a4364a213 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d584c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 778 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/2041cb76-6180-45d4-9ef3-91c6547c9ab9_calibrate_898798_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/2041cb76-6180-45d4-9ef3-91c6547c9ab9_calibrate_898798_15.stderr parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/4e347ece-ab75-47ab-9837-6522855f64d3_characterizeImage_898798_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/4e347ece-ab75-47ab-9837-6522855f64d3_characterizeImage_898798_15.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 352 try 1 failed parsl.dataflow.dflow ERROR: Task 352 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 77d7575d667b9ad34c090c8f1676287a with result from task 352 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 353 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 353 try 0 failed parsl.dataflow.dflow INFO: Task 353 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/02649c37-15d9-4ee1-a177-8773dd1c4dba_calibrate_898696_2.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/02649c37-15d9-4ee1-a177-8773dd1c4dba_calibrate_898696_2.stderr parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/645d169e-9684-4549-8e89-89431d6f7a50_characterizeImage_898696_2.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/645d169e-9684-4549-8e89-89431d6f7a50_characterizeImage_898696_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 457 try 1 failed parsl.dataflow.dflow ERROR: Task 457 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 1378e62d0a8735467f133c64ddf613fc with result from task 457 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 458 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 458 try 0 failed parsl.dataflow.dflow INFO: Task 458 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/83a93139-ee86-473d-900f-f276a17aa5b6_calibrate_898798_2.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/83a93139-ee86-473d-900f-f276a17aa5b6_calibrate_898798_2.stderr parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/b06bb2c5-1ca2-44e3-ba6d-40832489814f_characterizeImage_898798_2.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/b06bb2c5-1ca2-44e3-ba6d-40832489814f_characterizeImage_898798_2.stderr parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d53dd4499f3fd65a3315268deb72c954 with result from task 400 parsl.dataflow.dflow DEBUG: Task 400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 401 has memoization hash cd8c1c5510a32e1a6b7b055d199c80d8 parsl.dataflow.memoization INFO: Task 401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d662fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 779 parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/5988dc4c-4381-421a-9a10-7d9ceb990106_calibrate_898694_22.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/5988dc4c-4381-421a-9a10-7d9ceb990106_calibrate_898694_22.stderr parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/6ffe6d66-005b-4fa8-ade1-fd1a2fec53d7_characterizeImage_898694_22.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/6ffe6d66-005b-4fa8-ade1-fd1a2fec53d7_characterizeImage_898694_22.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d08c1ed2fcc405922fdaea9baa6e3549 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash eab6ea866cade56c2bd74e18793d9845 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ceb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 780 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/c19ec2df-8468-4b72-a23e-808108cf6f0d_calibrate_898800_15.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/c19ec2df-8468-4b72-a23e-808108cf6f0d_calibrate_898800_15.stderr parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/b534cb93-ef55-47d7-b3ce-2cd981eaad8c_characterizeImage_898800_15.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/b534cb93-ef55-47d7-b3ce-2cd981eaad8c_characterizeImage_898800_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:04 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5c936d22a68738f38ded50be702136d with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 3fc16c2fd561111d2ad00cd32bc28988 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ab2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 781 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/b8faf874-026c-4833-b1a8-628f0d710287_calibrate_898799_36.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/b8faf874-026c-4833-b1a8-628f0d710287_calibrate_898799_36.stderr parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/9ce11716-6ef1-4762-bf4e-4abd9244d1dd_characterizeImage_898799_36.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/9ce11716-6ef1-4762-bf4e-4abd9244d1dd_characterizeImage_898799_36.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f8cc9ed4058661154576f90c40ad5c1 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 0f054771da2eb5812e7803163d80b628 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cd260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 782 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/4f959a19-e0fb-4175-ba41-544090c91a89_calibrate_898695_35.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/4f959a19-e0fb-4175-ba41-544090c91a89_calibrate_898695_35.stderr parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/171067a5-07a8-4c61-bd48-e499acee32f6_characterizeImage_898695_35.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/171067a5-07a8-4c61-bd48-e499acee32f6_characterizeImage_898695_35.stderr parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 478a5105c55e59d175538fb08648c6a7 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash 12bd8f820466bb31a7c7fd6cd61854b0 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d586e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 783 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/257e67a7-46f5-42ef-89be-c3b94d53a720_calibrate_898800_36.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/257e67a7-46f5-42ef-89be-c3b94d53a720_calibrate_898800_36.stderr parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/549d0c03-35c4-47ae-a5cd-aef6687212a1_characterizeImage_898800_36.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/549d0c03-35c4-47ae-a5cd-aef6687212a1_characterizeImage_898800_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:06 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 868d3212c27a263337649cf308b17574 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 4456776c1a7e7755bbf4492452e4c880 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ece00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 784 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/04d52b69-6d52-4a59-a61f-27e65a749906_calibrate_898695_48.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/04d52b69-6d52-4a59-a61f-27e65a749906_calibrate_898695_48.stderr parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/3c4cd1f3-c8be-498e-a6ce-b81301f73d2c_characterizeImage_898695_48.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/3c4cd1f3-c8be-498e-a6ce-b81301f73d2c_characterizeImage_898695_48.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11566d03e09dfeff4ef92754904755c0 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 0204d2e3caa220a09f7e62701f95cca0 parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 785 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/9b54da11-a3b0-4f2d-9364-440f8c77abf0_calibrate_898696_41.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/9b54da11-a3b0-4f2d-9364-440f8c77abf0_calibrate_898696_41.stderr parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/258750c0-435e-4dea-8648-d0ef781f30ca_characterizeImage_898696_41.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/258750c0-435e-4dea-8648-d0ef781f30ca_characterizeImage_898696_41.stderr parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77df07d484b7fea77901ec1f476f1fbc with result from task 265 parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash 7ffa337185f55e3f82ee8395f69bf291 parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 786 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/adb2ab80-579f-4c08-b9b5-608add0b59b2_calibrate_898696_35.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/adb2ab80-579f-4c08-b9b5-608add0b59b2_calibrate_898696_35.stderr parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/c2929b31-c0de-491c-88ff-9ea32bdce3ed_characterizeImage_898696_35.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/c2929b31-c0de-491c-88ff-9ea32bdce3ed_characterizeImage_898696_35.stderr parsl.dataflow.dflow INFO: Task 484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0032946f3a2d3083b1ae655053c83a0c with result from task 484 parsl.dataflow.dflow DEBUG: Task 484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 485 has memoization hash f3effc77ce92a4ea844bb68ead9e15fd parsl.dataflow.memoization INFO: Task 485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4832e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 787 parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/29d2352d-173b-4550-af9d-10db86c150e2_calibrate_898696_31.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/29d2352d-173b-4550-af9d-10db86c150e2_calibrate_898696_31.stderr parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/558168ff-287a-471e-9027-1bce897f750f_characterizeImage_898696_31.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/558168ff-287a-471e-9027-1bce897f750f_characterizeImage_898696_31.stderr parsl.dataflow.dflow INFO: Task 535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de7aaa3378ffcf74ffdb971dee5ed00b with result from task 535 parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:07 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow DEBUG: Task 535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 536 has memoization hash 95c9df7e3467c9a45821c89d2151100e parsl.dataflow.memoization INFO: Task 536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dc0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 788 parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/67a34143-0a91-4d44-9a92-fbca4dc04eee_calibrate_898696_29.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/67a34143-0a91-4d44-9a92-fbca4dc04eee_calibrate_898696_29.stderr parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/2f8cd3fd-975c-43e8-aac9-a2e02be710b8_characterizeImage_898696_29.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/2f8cd3fd-975c-43e8-aac9-a2e02be710b8_characterizeImage_898696_29.stderr parsl.dataflow.dflow INFO: Task 466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd4d47c66a8dfd3660f44015c96bc6ec with result from task 466 parsl.dataflow.dflow DEBUG: Task 466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 467 has memoization hash de266c90ca88256a6c7428f5d58ca304 parsl.dataflow.memoization INFO: Task 467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 467 try 0 launched on executor multi with executor id 789 parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/451144f1-db7f-438b-b0ee-9914d5f7f7d6_calibrate_898695_10.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/451144f1-db7f-438b-b0ee-9914d5f7f7d6_calibrate_898695_10.stderr parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/176ae148-7e63-4550-ab80-ae100c2bdf2f_characterizeImage_898695_10.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/176ae148-7e63-4550-ab80-ae100c2bdf2f_characterizeImage_898695_10.stderr parsl.dataflow.dflow INFO: Task 439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b381f2df92226b0e4adbadb205c2fe70 with result from task 439 parsl.dataflow.dflow DEBUG: Task 439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 440 has memoization hash a8efd19bf852fa843fb4f84a1378aa1e parsl.dataflow.memoization INFO: Task 440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c481a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 440 try 0 launched on executor multi with executor id 790 parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/49fef6c9-bc54-45ab-a3ba-415bb62d5f15_calibrate_898800_47.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/49fef6c9-bc54-45ab-a3ba-415bb62d5f15_calibrate_898800_47.stderr parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/fdf0233a-36d7-4797-bf98-017281565e88_characterizeImage_898800_47.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/fdf0233a-36d7-4797-bf98-017281565e88_characterizeImage_898800_47.stderr parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90064832ab7598e07791d37e0e268956 with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash f31267956a209f55397bd2aa0c33dcb8 parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5eea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 791 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/346aefae-4916-4a50-861c-079de9b7154f_calibrate_898798_13.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/346aefae-4916-4a50-861c-079de9b7154f_calibrate_898798_13.stderr parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/c2ddae44-93f6-4afd-a19f-24f55955f00e_characterizeImage_898798_13.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/c2ddae44-93f6-4afd-a19f-24f55955f00e_characterizeImage_898798_13.stderr parsl.dataflow.dflow INFO: Task 607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 376be6dc4da82ff80e52f553bde9a7f4 with result from task 607 parsl.dataflow.dflow DEBUG: Task 607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 608 has memoization hash f8e8f9d3c1723b94c9afc197931fc6ef parsl.dataflow.memoization INFO: Task 608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c510c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 608 try 0 launched on executor multi with executor id 792 parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/77cbf1b1-54a5-4da0-8681-03aad7e9b23e_calibrate_898696_27.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/77cbf1b1-54a5-4da0-8681-03aad7e9b23e_calibrate_898696_27.stderr parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/ea912538-89ff-4622-87e2-ba0e5ae3f1aa_characterizeImage_898696_27.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/ea912538-89ff-4622-87e2-ba0e5ae3f1aa_characterizeImage_898696_27.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f76e6b3dcaeadf789603703ecdc426a6 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 02e61f810e09028ddb5ea5f2fd7a9ee5 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5871a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 793 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/086b15b4-5087-4884-bdc8-91ed27337297_calibrate_898799_24.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/086b15b4-5087-4884-bdc8-91ed27337297_calibrate_898799_24.stderr parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/ffe8e3ba-d958-4fcc-b3d0-c89100dfdbdf_characterizeImage_898799_24.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/ffe8e3ba-d958-4fcc-b3d0-c89100dfdbdf_characterizeImage_898799_24.stderr parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adba8059dd6abf10b89a5fe5a6faf3bf with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 8c2bb2f0a43e8dee9513dd183f6b2c34 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5aa340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 794 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/5ebab2f8-b8be-4192-8a00-1e8d50ec3465_calibrate_898798_52.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/5ebab2f8-b8be-4192-8a00-1e8d50ec3465_calibrate_898798_52.stderr parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/694194a0-035a-4778-93c2-7d66b7c57abc_characterizeImage_898798_52.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/694194a0-035a-4778-93c2-7d66b7c57abc_characterizeImage_898798_52.stderr parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5dd3e2eefffd04cc870b7f4796578526 with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash 854d45734defbcce16279b91fc63eb99 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ee700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 795 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/028e8432-65fb-4d28-89ee-faac0ef2df8a_calibrate_898799_8.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/028e8432-65fb-4d28-89ee-faac0ef2df8a_calibrate_898799_8.stderr parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/d35048fd-afbe-4dae-b758-499d05ab9fbd_characterizeImage_898799_8.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/d35048fd-afbe-4dae-b758-499d05ab9fbd_characterizeImage_898799_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:08 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 257894ad9702ea584eb6a457551643cd with result from task 652 parsl.dataflow.dflow DEBUG: Task 652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 653 has memoization hash 3cb7b1cb61161f4cc4b9f0840c80bca1 parsl.dataflow.memoization INFO: Task 653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c5139c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 653 try 0 launched on executor multi with executor id 796 parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/4c41eb90-e342-4e61-b835-6426f7d01235_calibrate_898696_40.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/4c41eb90-e342-4e61-b835-6426f7d01235_calibrate_898696_40.stderr parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/a6b856ac-9728-474b-a870-462982feb8d5_characterizeImage_898696_40.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/a6b856ac-9728-474b-a870-462982feb8d5_characterizeImage_898696_40.stderr parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d42d0891d60c5f3b2847504398757b2 with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash 5fb075befdc4b131cc1724dacd60ad2f parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 797 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/45da7648-702b-4c46-8d8e-6cc39fe8e8fa_calibrate_898694_3.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/45da7648-702b-4c46-8d8e-6cc39fe8e8fa_calibrate_898694_3.stderr parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/1429c4df-06f9-4ff7-9a76-fda914202966_characterizeImage_898694_3.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/1429c4df-06f9-4ff7-9a76-fda914202966_characterizeImage_898694_3.stderr parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02989870e1b241773fa38c36b1b9756a with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash 867a43e62a422db1171e0b8bf03f0dc7 parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ef9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 798 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/08d52931-8a23-49d2-8887-c0f474a6f107_calibrate_898800_12.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/08d52931-8a23-49d2-8887-c0f474a6f107_calibrate_898800_12.stderr parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/3c13976b-d5a2-44b3-9c24-8c2a848a4d9d_characterizeImage_898800_12.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/3c13976b-d5a2-44b3-9c24-8c2a848a4d9d_characterizeImage_898800_12.stderr parsl.dataflow.dflow INFO: Task 454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d7de60cf1170360143eea50b8708ba0 with result from task 454 parsl.dataflow.dflow DEBUG: Task 454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 455 has memoization hash 9375e9a95e8ddc1abaa1ec3bf49fdc99 parsl.dataflow.memoization INFO: Task 455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 799 parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/01ba1150-bde7-4069-a60c-67ca1472d3a6_calibrate_898696_48.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/01ba1150-bde7-4069-a60c-67ca1472d3a6_calibrate_898696_48.stderr parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/c1a5c801-fc27-4796-b92e-fe15017c2969_characterizeImage_898696_48.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/c1a5c801-fc27-4796-b92e-fe15017c2969_characterizeImage_898696_48.stderr parsl.dataflow.dflow INFO: Task 865 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63d4f222da1c4d65b60b79252ba66af6 with result from task 865 parsl.dataflow.dflow DEBUG: Task 865 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 866 has memoization hash c7ad7eef27babb89a6036b85d853b31b parsl.dataflow.memoization INFO: Task 866 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4419e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 866 try 0 launched on executor multi with executor id 800 parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/36cd6351-d8f1-4f8b-9c37-4697f068f819_calibrate_898694_30.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/36cd6351-d8f1-4f8b-9c37-4697f068f819_calibrate_898694_30.stderr parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/8b08f397-dc5f-4f99-ae24-49710aa06755_characterizeImage_898694_30.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/8b08f397-dc5f-4f99-ae24-49710aa06755_characterizeImage_898694_30.stderr parsl.dataflow.dflow INFO: Task 604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5f9493cbb9a6dc0aee9f32409c4b3d2 with result from task 604 parsl.dataflow.dflow DEBUG: Task 604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 605 has memoization hash 8ba56cdb911350dd74ef28deb65a0145 parsl.dataflow.memoization INFO: Task 605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c510900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 605 try 0 launched on executor multi with executor id 801 parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/097c61cd-a697-442b-973e-dcf72dd6983b_calibrate_898800_51.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/097c61cd-a697-442b-973e-dcf72dd6983b_calibrate_898800_51.stderr parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/297a28a5-09c7-43b1-b327-0d2bcfa45fa6_characterizeImage_898800_51.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/297a28a5-09c7-43b1-b327-0d2bcfa45fa6_characterizeImage_898800_51.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c551798e220602c838de58b53d28be7 with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 1dda491908ea52945d6b233d707a661b parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5874c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 802 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/a225bc31-deb5-4bb2-a6b4-7c3e46e8beeb_calibrate_898696_18.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/a225bc31-deb5-4bb2-a6b4-7c3e46e8beeb_calibrate_898696_18.stderr parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/bbd81449-432e-4786-bd8e-4c5934eed9e5_characterizeImage_898696_18.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/bbd81449-432e-4786-bd8e-4c5934eed9e5_characterizeImage_898696_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:10 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e53527e76b669dc9611cb7fae37ee03 with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash ac375c17effe10dca2266b8530606932 parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 803 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/77b16008-00f8-40a3-a09f-bac90807ffb3_calibrate_898695_58.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/77b16008-00f8-40a3-a09f-bac90807ffb3_calibrate_898695_58.stderr parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/0f78ee3a-dc59-4311-9169-1e338a576584_characterizeImage_898695_58.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/0f78ee3a-dc59-4311-9169-1e338a576584_characterizeImage_898695_58.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5346155ea9c7580051a40bb1ef4eaaac with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash b4b1a667e668c726892ae0ac6e4463c1 parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ec7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 804 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/f7ff8c72-b0c8-4985-a73a-658c4c8ecb8e_calibrate_898800_24.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/f7ff8c72-b0c8-4985-a73a-658c4c8ecb8e_calibrate_898800_24.stderr parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/73c4918a-187f-4be1-9611-bd1abfc5ccb8_characterizeImage_898800_24.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/73c4918a-187f-4be1-9611-bd1abfc5ccb8_characterizeImage_898800_24.stderr parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14d4c28bb5ebde2568fff6e781bfaa3f with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash 6211df7411ed63918753a11b8bc62405 parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d609f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 805 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/20f9ed90-1f9d-46ba-a4ae-ecb5588576eb_calibrate_898798_54.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/20f9ed90-1f9d-46ba-a4ae-ecb5588576eb_calibrate_898798_54.stderr parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/16aad049-f9e6-47c1-b997-b61c2f7884a1_characterizeImage_898798_54.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/16aad049-f9e6-47c1-b997-b61c2f7884a1_characterizeImage_898798_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:11 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc28a2e33c42f5bc028f5215051f8427 with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 62656e040e037596f3c592f165e6598f parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ce520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 806 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/32b0ead4-705a-4191-b922-12f6a784a892_calibrate_898695_17.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/32b0ead4-705a-4191-b922-12f6a784a892_calibrate_898695_17.stderr parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/e8799074-2fd1-4814-868b-79ec584004cf_characterizeImage_898695_17.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/e8799074-2fd1-4814-868b-79ec584004cf_characterizeImage_898695_17.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66bf19c617b9f95b214636c4cdf757b9 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 9253716e473646433c98bf9bc5e8e2f2 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cf4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 807 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/cdc21c34-7dec-4f62-8dca-3aca29b7bb87_calibrate_898695_51.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/cdc21c34-7dec-4f62-8dca-3aca29b7bb87_calibrate_898695_51.stderr parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/2ec118bb-c4df-4ff4-b396-0b8c250eb660_characterizeImage_898695_51.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/2ec118bb-c4df-4ff4-b396-0b8c250eb660_characterizeImage_898695_51.stderr parsl.dataflow.dflow INFO: Task 982 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41d455a6bec999c126225bcea9ca351c with result from task 982 parsl.dataflow.dflow DEBUG: Task 982 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 983 has memoization hash 50613e85f53185aa7a85d72305d99e23 parsl.dataflow.memoization INFO: Task 983 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 983 try 0 launched on executor multi with executor id 808 parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/2af6388d-fd7f-47af-85cd-226d4e4c6c34_calibrate_898696_34.stdout parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/2af6388d-fd7f-47af-85cd-226d4e4c6c34_calibrate_898696_34.stderr parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/ccc294d5-d3bd-499f-b185-ffc22508a410_characterizeImage_898696_34.stdout parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/ccc294d5-d3bd-499f-b185-ffc22508a410_characterizeImage_898696_34.stderr parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8690543670a892931ddc48e9e73dae1 with result from task 241 parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:12 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 4410040e7b00b77e57b0334b160560bd parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6089a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 809 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/6e27a573-1852-481d-9526-62ec97e73f0b_calibrate_898800_18.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/6e27a573-1852-481d-9526-62ec97e73f0b_calibrate_898800_18.stderr parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/21f27cea-9879-4c0f-93d6-d1586feca95f_characterizeImage_898800_18.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/21f27cea-9879-4c0f-93d6-d1586feca95f_characterizeImage_898800_18.stderr parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 810ff313ba075f5452e9ee35aa7cba07 with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 8265c41aa4137cdcb4b6d672f8ffc5c8 parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 810 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/77d03706-989e-4cd6-a607-20a478d9fac2_calibrate_898694_54.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/77d03706-989e-4cd6-a607-20a478d9fac2_calibrate_898694_54.stderr parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/ad417399-876c-4066-8c82-f14945c7fd0a_characterizeImage_898694_54.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/ad417399-876c-4066-8c82-f14945c7fd0a_characterizeImage_898694_54.stderr parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7235efac539c7de40a3750651ce9588 with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 7e05f1dcc8768e91a19cf0525c5e3d16 parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5edda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 811 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/716eb7c7-b003-47c9-8f11-ca0a533e2dd4_calibrate_898695_18.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/716eb7c7-b003-47c9-8f11-ca0a533e2dd4_calibrate_898695_18.stderr parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/b30939dc-db14-44f9-838e-a275d60ac4c0_characterizeImage_898695_18.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/b30939dc-db14-44f9-838e-a275d60ac4c0_characterizeImage_898695_18.stderr parsl.dataflow.dflow INFO: Task 382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a412a3eba52b2afe3afe08a7c995c9e with result from task 382 parsl.dataflow.dflow DEBUG: Task 382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 383 has memoization hash 365cad3295f57dd9b2d7ef43d1bbcdc7 parsl.dataflow.memoization INFO: Task 383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d661d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 812 parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/e71634c4-a3b4-4248-9afa-22294ed75443_calibrate_898800_59.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/e71634c4-a3b4-4248-9afa-22294ed75443_calibrate_898800_59.stderr parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/442f0996-5a0c-4f0d-b72f-c1a0f0573954_characterizeImage_898800_59.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/442f0996-5a0c-4f0d-b72f-c1a0f0573954_characterizeImage_898800_59.stderr parsl.dataflow.dflow INFO: Task 514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfa6e832111110eba0523c4ca978853b with result from task 514 parsl.dataflow.dflow DEBUG: Task 514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 515 has memoization hash f584ec7d3d7e83ff04cc018c61c69006 parsl.dataflow.memoization INFO: Task 515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 813 parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/50df710f-f03e-4478-b1d2-39a06b0154f2_calibrate_898696_53.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/50df710f-f03e-4478-b1d2-39a06b0154f2_calibrate_898696_53.stderr parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/971f994b-3f52-471c-890a-7a4badb5fdc8_characterizeImage_898696_53.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/971f994b-3f52-471c-890a-7a4badb5fdc8_characterizeImage_898696_53.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d847158c2f7d8470b2fb6470e9e3f5a2 with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 4c2d45e0d216a520c0db928b962c0a06 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ecae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 814 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/b17b7acc-82d4-4d33-a96b-11afc3600de8_calibrate_898800_41.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/b17b7acc-82d4-4d33-a96b-11afc3600de8_calibrate_898800_41.stderr parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/a8ad69e8-5f8e-4c21-87f9-27f0ec9238a0_characterizeImage_898800_41.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/a8ad69e8-5f8e-4c21-87f9-27f0ec9238a0_characterizeImage_898800_41.stderr parsl.dataflow.dflow INFO: Task 532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8258377d21655f9632bedb56f4966a36 with result from task 532 parsl.dataflow.dflow DEBUG: Task 532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 533 has memoization hash 0df90aad7be31b1c901826edd3109102 parsl.dataflow.memoization INFO: Task 533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 533 try 0 launched on executor multi with executor id 815 parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/d95b67b1-6d86-4a04-8a59-839315d8e6f5_calibrate_898695_31.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/d95b67b1-6d86-4a04-8a59-839315d8e6f5_calibrate_898695_31.stderr parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/c25fe78b-493a-444c-9362-81318b35066f_characterizeImage_898695_31.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/c25fe78b-493a-444c-9362-81318b35066f_characterizeImage_898695_31.stderr parsl.dataflow.dflow INFO: Task 415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2546251b30857eab03dad54061293d7a with result from task 415 parsl.dataflow.dflow DEBUG: Task 415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 416 has memoization hash d811da4aca86e871d056b859621d9ee5 parsl.dataflow.memoization INFO: Task 416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c480040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 416 try 0 launched on executor multi with executor id 816 parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/45407b46-ecba-485a-aba9-80d0de53d9f8_calibrate_898695_46.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/45407b46-ecba-485a-aba9-80d0de53d9f8_calibrate_898695_46.stderr parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/1f61806b-9e10-4b93-ab52-5f1eb8e89227_characterizeImage_898695_46.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/1f61806b-9e10-4b93-ab52-5f1eb8e89227_characterizeImage_898695_46.stderr parsl.dataflow.dflow INFO: Task 1000 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 369a19cd5ce86588e60a2ed3bb5d6db5 with result from task 1000 parsl.dataflow.dflow DEBUG: Task 1000 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1001 has memoization hash 515f0ef9e8ba556be5b6f3811deb20e1 parsl.dataflow.memoization INFO: Task 1001 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2da700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1001 try 0 launched on executor multi with executor id 817 parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/e3fb9fc1-9d42-4db8-a913-c5204576f223_calibrate_898695_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/e3fb9fc1-9d42-4db8-a913-c5204576f223_calibrate_898695_42.stderr parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/e1da4da7-ed11-492a-9c04-e1d2a35b411a_characterizeImage_898695_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/e1da4da7-ed11-492a-9c04-e1d2a35b411a_characterizeImage_898695_42.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7781801a28e04d6de4dd99e0af521968 with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash a9282eaedd3e8b9d5eb3e59082a84390 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cf7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 818 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/f4952490-d9f6-47cd-9093-5f1e74b5d43c_calibrate_898800_27.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/f4952490-d9f6-47cd-9093-5f1e74b5d43c_calibrate_898800_27.stderr parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/8c3c95d4-cc33-475d-bae3-0a55f6b85477_characterizeImage_898800_27.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/8c3c95d4-cc33-475d-bae3-0a55f6b85477_characterizeImage_898800_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:14 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c2b19577eb89e0c3554fba8314925b7 with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash 09b010cf0e36aa1eed04cbddb03f249d parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d661080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 819 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/9e3b741c-2923-4e7f-a128-b478e0ef6a0c_calibrate_898696_38.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/9e3b741c-2923-4e7f-a128-b478e0ef6a0c_calibrate_898696_38.stderr parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/5463ca1b-cad7-4ff8-a86c-3c0f15066055_characterizeImage_898696_38.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/5463ca1b-cad7-4ff8-a86c-3c0f15066055_characterizeImage_898696_38.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e94b93d0373c690e2ecccdfcd91ca3e with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash f5b0aa74853c0619464fd38b0c354ed1 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cd8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 820 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/93db3b8d-71fe-434a-b80f-acb432cdad03_calibrate_898695_25.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/93db3b8d-71fe-434a-b80f-acb432cdad03_calibrate_898695_25.stderr parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/6c4ab54e-98ef-4232-a91d-c83df954df4b_characterizeImage_898695_25.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/6c4ab54e-98ef-4232-a91d-c83df954df4b_characterizeImage_898695_25.stderr parsl.dataflow.dflow INFO: Task 748 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfbb96fa1ba846f8ba31b9ea8b13087e with result from task 748 parsl.dataflow.dflow DEBUG: Task 748 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 749 has memoization hash 511d3187a6d7179f2a5b4a6a1cb38246 parsl.dataflow.memoization INFO: Task 749 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 749 try 0 launched on executor multi with executor id 821 parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/72461243-3334-4ad4-bc69-93861e1a3053_calibrate_898696_62.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/72461243-3334-4ad4-bc69-93861e1a3053_calibrate_898696_62.stderr parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/5ea0781b-7114-4dc9-81a4-ebbb7c1db901_characterizeImage_898696_62.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/5ea0781b-7114-4dc9-81a4-ebbb7c1db901_characterizeImage_898696_62.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b1aa58ad951726c309b90013aec2ec4 with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash ef2ecd6e52ac1b7cfa9da0579684da9e parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 822 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/cdf98ae9-1ed9-4b9d-b2db-94bfab1cd3a7_calibrate_898694_46.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/cdf98ae9-1ed9-4b9d-b2db-94bfab1cd3a7_calibrate_898694_46.stderr parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/a30b4722-d78f-467e-ab68-6bb6c9b9c5cc_characterizeImage_898694_46.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/a30b4722-d78f-467e-ab68-6bb6c9b9c5cc_characterizeImage_898694_46.stderr parsl.dataflow.dflow INFO: Task 1078 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f065869e58ceec1d7993ba69a9eeab3e with result from task 1078 parsl.dataflow.dflow DEBUG: Task 1078 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1079 has memoization hash bb07ab0caafa9863a2e9259e145f5de9 parsl.dataflow.memoization INFO: Task 1079 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c329c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1079 try 0 launched on executor multi with executor id 823 parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/25565815-0fe7-464d-9715-2b55b455551c_calibrate_898696_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/25565815-0fe7-464d-9715-2b55b455551c_calibrate_898696_58.stderr parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/57d27afa-c37e-4f64-8797-f29d3b5ee3e9_characterizeImage_898696_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/57d27afa-c37e-4f64-8797-f29d3b5ee3e9_characterizeImage_898696_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:15 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7092cc857ba5385dc97a22ddc8b5d1b with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash ee92661a16c310535ac25f5031de31e7 parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 824 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/95f04615-b52b-4ebc-b505-b2cf4516cd8b_calibrate_898799_52.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/95f04615-b52b-4ebc-b505-b2cf4516cd8b_calibrate_898799_52.stderr parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/59a24470-1455-4a8c-95ac-b228e51dc40b_characterizeImage_898799_52.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/59a24470-1455-4a8c-95ac-b228e51dc40b_characterizeImage_898799_52.stderr parsl.dataflow.dflow INFO: Task 463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de9e54deccef1d8d46dc2a788b32138f with result from task 463 parsl.dataflow.dflow DEBUG: Task 463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 464 has memoization hash 867d4493385f02c14877c41ddd3f7238 parsl.dataflow.memoization INFO: Task 464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 464 try 0 launched on executor multi with executor id 825 parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/f7eb1141-9965-4791-999d-f615d2a9b0bb_calibrate_898798_56.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/f7eb1141-9965-4791-999d-f615d2a9b0bb_calibrate_898798_56.stderr parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/727ab573-8ad1-437d-94f3-a33f49b25ff6_characterizeImage_898798_56.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/727ab573-8ad1-437d-94f3-a33f49b25ff6_characterizeImage_898798_56.stderr parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8793d37da229a72a703b1c7ada078c5 with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 6c68f2fcac0cbdef7d51ef39f5a003d1 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 826 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/c4697bb9-d8b6-4cf8-b0e7-0d1e0c95777c_calibrate_898798_24.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/c4697bb9-d8b6-4cf8-b0e7-0d1e0c95777c_calibrate_898798_24.stderr parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/33acfb60-5306-4ee5-a475-63de6bd27afb_characterizeImage_898798_24.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/33acfb60-5306-4ee5-a475-63de6bd27afb_characterizeImage_898798_24.stderr parsl.dataflow.dflow INFO: Task 727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b18332daa6130cfbec090e98f7f02611 with result from task 727 parsl.dataflow.dflow DEBUG: Task 727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 728 has memoization hash 371d54f637f676ab93b63efc085ffbc1 parsl.dataflow.memoization INFO: Task 728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 728 try 0 launched on executor multi with executor id 827 parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/42f7c779-c4ce-45ac-9e7d-8855f618f8ae_calibrate_898695_57.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/42f7c779-c4ce-45ac-9e7d-8855f618f8ae_calibrate_898695_57.stderr parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/f537cbd5-a625-45ec-a865-8afb2080d283_characterizeImage_898695_57.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/f537cbd5-a625-45ec-a865-8afb2080d283_characterizeImage_898695_57.stderr parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e4bb11b3f96392e583635adf44c82e4 with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash b01595cdde73f2d669272c99bdd45e61 parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 828 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/4301f74b-a89c-462e-b16c-49c2e6af82f0_calibrate_898694_1.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/4301f74b-a89c-462e-b16c-49c2e6af82f0_calibrate_898694_1.stderr parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/db1c13f1-cdff-4920-a37c-97dcc9c56500_characterizeImage_898694_1.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/db1c13f1-cdff-4920-a37c-97dcc9c56500_characterizeImage_898694_1.stderr parsl.dataflow.dflow INFO: Task 979 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e03a77d893589d9c95913d19cfec40b3 with result from task 979 parsl.dataflow.dflow DEBUG: Task 979 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 980 has memoization hash d05b90e53297be3c3b28b18e2ee713a9 parsl.dataflow.memoization INFO: Task 980 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 980 try 0 launched on executor multi with executor id 829 parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/c1e96afe-ae4e-4941-b2ad-d7e802f0b5de_calibrate_898800_21.stdout parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/c1e96afe-ae4e-4941-b2ad-d7e802f0b5de_calibrate_898800_21.stderr parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/81ba27f1-29c9-43f4-bd71-376068c42405_characterizeImage_898800_21.stdout parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/81ba27f1-29c9-43f4-bd71-376068c42405_characterizeImage_898800_21.stderr parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 307e0af186aa73805d3f884933b67d6e with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash e6fba352a342b727deded0e34e29e2fc parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d608360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 830 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/6699c441-8c53-4e59-ae93-cc5bebfd31b1_calibrate_898799_42.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/6699c441-8c53-4e59-ae93-cc5bebfd31b1_calibrate_898799_42.stderr parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/2d9daff9-a0d6-413e-928c-96561de15d71_characterizeImage_898799_42.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/2d9daff9-a0d6-413e-928c-96561de15d71_characterizeImage_898799_42.stderr parsl.dataflow.dflow INFO: Task 574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 741fd3767734dde779ed85e462b7da01 with result from task 574 parsl.dataflow.dflow DEBUG: Task 574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 575 has memoization hash c119897be80a5ccec9c3b2e8836c5f34 parsl.dataflow.memoization INFO: Task 575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4de2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 575 try 0 launched on executor multi with executor id 831 parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/130f6928-d5e3-4892-95a5-2e78577a269f_calibrate_898798_20.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/130f6928-d5e3-4892-95a5-2e78577a269f_calibrate_898798_20.stderr parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/fa3ca50b-e138-4ee0-81eb-606a4f103cb3_characterizeImage_898798_20.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/fa3ca50b-e138-4ee0-81eb-606a4f103cb3_characterizeImage_898798_20.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e497cba56f4d44e443f6491ae4f16f25 with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 3deb883ad37102bd292b01485d0a03f1 parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ee0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 832 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/69234db1-835a-4f65-bd43-fe502c9e482e_calibrate_898694_5.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/69234db1-835a-4f65-bd43-fe502c9e482e_calibrate_898694_5.stderr parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/57e3faad-3c33-4bf4-9483-c00249722cb9_characterizeImage_898694_5.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/57e3faad-3c33-4bf4-9483-c00249722cb9_characterizeImage_898694_5.stderr parsl.dataflow.dflow INFO: Task 523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8057a1e552cebddd28e3a36ba08d47f5 with result from task 523 parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:16 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow DEBUG: Task 523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 524 has memoization hash 095845982a337b72a6df93ee17554244 parsl.dataflow.memoization INFO: Task 524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 524 try 0 launched on executor multi with executor id 833 parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/05cefb02-a578-401b-8150-e140b399934c_calibrate_898696_10.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/05cefb02-a578-401b-8150-e140b399934c_calibrate_898696_10.stderr parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/069409b1-6cf8-428a-9db6-be9f2eebf5ae_characterizeImage_898696_10.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/069409b1-6cf8-428a-9db6-be9f2eebf5ae_characterizeImage_898696_10.stderr parsl.dataflow.dflow INFO: Task 892 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c954498ab32a2ec5b3d30ea5ddbe8da1 with result from task 892 parsl.dataflow.dflow DEBUG: Task 892 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 893 has memoization hash 1a5518d69f95a24a0fcc8a0af8e615b7 parsl.dataflow.memoization INFO: Task 893 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c443600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 893 try 0 launched on executor multi with executor id 834 parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/d9565533-4f3f-4fc8-853d-13085f870ac3_calibrate_898798_59.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/d9565533-4f3f-4fc8-853d-13085f870ac3_calibrate_898798_59.stderr parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/88975824-e18e-4e1b-89f3-d213b68b672f_characterizeImage_898798_59.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/88975824-e18e-4e1b-89f3-d213b68b672f_characterizeImage_898798_59.stderr parsl.dataflow.dflow INFO: Task 772 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b679c08d61608d76981c3ded207952b with result from task 772 parsl.dataflow.dflow DEBUG: Task 772 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 773 has memoization hash ff034a870caaf7870aecb2c14bd89dd0 parsl.dataflow.memoization INFO: Task 773 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 773 try 0 launched on executor multi with executor id 835 parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/df617131-49f6-42eb-a49c-d0ce8ac8efd7_calibrate_898695_7.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/df617131-49f6-42eb-a49c-d0ce8ac8efd7_calibrate_898695_7.stderr parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/761e1ce1-11ef-406b-9922-fc27fa7c8ff7_characterizeImage_898695_7.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/761e1ce1-11ef-406b-9922-fc27fa7c8ff7_characterizeImage_898695_7.stderr parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b78d2e12ebfb25511c61ad4f76b9a5f with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash b05b9a13af81bb2c244eb6b67459e244 parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ef060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 836 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/8bffb5ee-161d-4d60-aa3d-15894630a60e_calibrate_898799_23.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/8bffb5ee-161d-4d60-aa3d-15894630a60e_calibrate_898799_23.stderr parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/082c8ade-5624-4fbb-8091-aeac9edc6616_characterizeImage_898799_23.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/082c8ade-5624-4fbb-8091-aeac9edc6616_characterizeImage_898799_23.stderr parsl.dataflow.dflow INFO: Task 667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3975a35ad57dae249c42f475313db75a with result from task 667 parsl.dataflow.dflow DEBUG: Task 667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 668 has memoization hash 61f539d1e6e3519a9dd371a0bd0780f2 parsl.dataflow.memoization INFO: Task 668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c374180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 668 try 0 launched on executor multi with executor id 837 parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/ca1954fe-6137-47d8-ad1b-0c3212daa9f0_calibrate_898800_11.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/ca1954fe-6137-47d8-ad1b-0c3212daa9f0_calibrate_898800_11.stderr parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/6a282091-7d72-4dd6-b8c0-3517d70f0f1e_characterizeImage_898800_11.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/6a282091-7d72-4dd6-b8c0-3517d70f0f1e_characterizeImage_898800_11.stderr parsl.dataflow.dflow INFO: Task 640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5defc5391d5b5dac2bd71eb2394a80a7 with result from task 640 parsl.dataflow.dflow DEBUG: Task 640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 641 has memoization hash af39f386bb93bca2cea42e0de504dbfd parsl.dataflow.memoization INFO: Task 641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c512e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 641 try 0 launched on executor multi with executor id 838 parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/191cc46a-bf97-4846-9105-a64484e049e6_calibrate_898799_62.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/191cc46a-bf97-4846-9105-a64484e049e6_calibrate_898799_62.stderr parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/5dd416c4-9f42-4d0d-92d7-b499c434cf66_characterizeImage_898799_62.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/5dd416c4-9f42-4d0d-92d7-b499c434cf66_characterizeImage_898799_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:18 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 1084 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4666dce1da757b44e18bb3346b7e556c with result from task 1084 parsl.dataflow.dflow DEBUG: Task 1084 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1085 has memoization hash 76ec9a6ccceff9c95590c04eadc918ca parsl.dataflow.memoization INFO: Task 1085 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1085 try 0 launched on executor multi with executor id 839 parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/4e463847-5972-4270-905f-3a83caba0df4_calibrate_898695_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/4e463847-5972-4270-905f-3a83caba0df4_calibrate_898695_37.stderr parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/3b44af96-4c82-455e-822e-d93ffa466e67_characterizeImage_898695_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/3b44af96-4c82-455e-822e-d93ffa466e67_characterizeImage_898695_37.stderr parsl.dataflow.dflow INFO: Task 871 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c36aed07051d255aa6581eff73475106 with result from task 871 parsl.dataflow.dflow DEBUG: Task 871 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 872 has memoization hash 5851da1157bd08528b28aa72cda97119 parsl.dataflow.memoization INFO: Task 872 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c442020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 872 try 0 launched on executor multi with executor id 840 parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/b92560b7-c235-42ab-b3a3-275bc4c2a203_calibrate_898696_54.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/b92560b7-c235-42ab-b3a3-275bc4c2a203_calibrate_898696_54.stderr parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/95fd5844-b9eb-40d2-b532-9daf59eab2b8_characterizeImage_898696_54.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/95fd5844-b9eb-40d2-b532-9daf59eab2b8_characterizeImage_898696_54.stderr parsl.dataflow.dflow INFO: Task 481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e9f44d14cdb055df20921d790fa5131 with result from task 481 parsl.dataflow.dflow DEBUG: Task 481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 482 has memoization hash 5076ae32905a210b7b375fe9a967e6df parsl.dataflow.memoization INFO: Task 482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c482fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 482 try 0 launched on executor multi with executor id 841 parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/3372209f-37c4-45aa-aa61-3a0ca245cc2e_calibrate_898799_59.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/3372209f-37c4-45aa-aa61-3a0ca245cc2e_calibrate_898799_59.stderr parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/babf7d0c-1098-4c39-88ef-cd5f18f1cb48_characterizeImage_898799_59.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/babf7d0c-1098-4c39-88ef-cd5f18f1cb48_characterizeImage_898799_59.stderr parsl.dataflow.dflow INFO: Task 796 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 058d09ae509dcbd9f5b038b7e5eef197 with result from task 796 parsl.dataflow.dflow DEBUG: Task 796 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 797 has memoization hash f9ed2aaa60daaea70de57cb638421589 parsl.dataflow.memoization INFO: Task 797 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3bbec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 797 try 0 launched on executor multi with executor id 842 parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/d5f5e494-ab7c-4c87-9683-1f7330397e0e_calibrate_898798_36.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/d5f5e494-ab7c-4c87-9683-1f7330397e0e_calibrate_898798_36.stderr parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/affa5b59-d1c0-4830-a90d-ef67ec6d5796_characterizeImage_898798_36.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/affa5b59-d1c0-4830-a90d-ef67ec6d5796_characterizeImage_898798_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:19 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a826b2acb9732fd7f36988e976647d8c with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 291aa7f4a8b6e8db89cedde669f07141 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d585580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 843 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/f05034f5-bf2c-4425-9c65-2a5244c29081_calibrate_898798_6.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/f05034f5-bf2c-4425-9c65-2a5244c29081_calibrate_898798_6.stderr parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/b687ec28-03c2-4c23-8d95-d9792c516218_characterizeImage_898798_6.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/b687ec28-03c2-4c23-8d95-d9792c516218_characterizeImage_898798_6.stderr parsl.dataflow.dflow INFO: Task 1006 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f30475a05cbb57856fd3b167428f8bb9 with result from task 1006 parsl.dataflow.dflow DEBUG: Task 1006 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1007 has memoization hash 9afe1f5cec199e7e4c002b83ab4c1420 parsl.dataflow.memoization INFO: Task 1007 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2dad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1007 try 0 launched on executor multi with executor id 844 parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/f3ae9237-0dd3-4017-a2b2-b6c3d541f6a8_calibrate_898695_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/f3ae9237-0dd3-4017-a2b2-b6c3d541f6a8_calibrate_898695_38.stderr parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/17fe7c0d-7f90-4928-a64d-d2775725ddde_characterizeImage_898695_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/17fe7c0d-7f90-4928-a64d-d2775725ddde_characterizeImage_898695_38.stderr parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50e9032ebad964bc9f2bbaa6b607549f with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash 027ef5c42fe821817ec75486d682b0df parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 845 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/485fa54b-f5db-4b1d-b27c-21eb344a00bb_calibrate_898694_58.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/485fa54b-f5db-4b1d-b27c-21eb344a00bb_calibrate_898694_58.stderr parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/6154a904-5398-4d5e-b3b0-b68ca859b50d_characterizeImage_898694_58.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/6154a904-5398-4d5e-b3b0-b68ca859b50d_characterizeImage_898694_58.stderr parsl.dataflow.dflow INFO: Task 1093 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 976612c3e713d07af1648344ea425d7a with result from task 1093 parsl.dataflow.dflow DEBUG: Task 1093 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1094 has memoization hash 978f6504e0ae94da82a75940bbf7911f parsl.dataflow.memoization INFO: Task 1094 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c350860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1094 try 0 launched on executor multi with executor id 846 parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/392ca369-d0cd-446b-aed2-2dba357e4d1d_calibrate_898694_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/392ca369-d0cd-446b-aed2-2dba357e4d1d_calibrate_898694_12.stderr parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/b5db42d4-bde8-40d1-8b21-57ac01a24056_characterizeImage_898694_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/b5db42d4-bde8-40d1-8b21-57ac01a24056_characterizeImage_898694_12.stderr parsl.dataflow.dflow INFO: Task 784 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36d944b72292755f4332cbd7202482fb with result from task 784 parsl.dataflow.dflow DEBUG: Task 784 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 785 has memoization hash 9fc34ecad3f61c92496f065eb9b69d1d parsl.dataflow.memoization INFO: Task 785 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3bb380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 785 try 0 launched on executor multi with executor id 847 parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/634f47cd-bf7c-4d66-956d-a1072b322885_calibrate_898696_32.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/634f47cd-bf7c-4d66-956d-a1072b322885_calibrate_898696_32.stderr parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/77515ba1-4685-4b79-b3fd-01a10dd7da13_characterizeImage_898696_32.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/77515ba1-4685-4b79-b3fd-01a10dd7da13_characterizeImage_898696_32.stderr parsl.dataflow.dflow INFO: Task 835 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f389e73fa0cf0a5fbebd01e51dfe495 with result from task 835 parsl.dataflow.dflow DEBUG: Task 835 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 836 has memoization hash cee341ccb4e9c3a0221b38837164dc80 parsl.dataflow.memoization INFO: Task 836 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 836 try 0 launched on executor multi with executor id 848 parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/f07dd838-2b9f-4c84-b6a4-07f3351cd576_calibrate_898799_40.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/f07dd838-2b9f-4c84-b6a4-07f3351cd576_calibrate_898799_40.stderr parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/23c8dcc2-7993-4e10-9c73-f65123cca940_characterizeImage_898799_40.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/23c8dcc2-7993-4e10-9c73-f65123cca940_characterizeImage_898799_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:20 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:21 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f5dbd0065a9fe5fbdb455195b5def88 with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash d0cd142090f1d2b63b8c7027eaeaca68 parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 849 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/2133ade3-9813-4daf-b0be-7a3221a65e69_calibrate_898799_22.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/2133ade3-9813-4daf-b0be-7a3221a65e69_calibrate_898799_22.stderr parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/caadaaf8-6f9e-4b48-b49b-2337836a1003_characterizeImage_898799_22.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/caadaaf8-6f9e-4b48-b49b-2337836a1003_characterizeImage_898799_22.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed1f0b65c5a699a3f496da5e186f8b4c with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/1518567d-1ff8-49c7-bee9-3513c642630d_calibrate_898694_31.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/1518567d-1ff8-49c7-bee9-3513c642630d_calibrate_898694_31.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3885b107a5995ab1810bef4bdb83e233 with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash fd398edf1a1f39243c5b84cf2ac46151 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ec4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 850 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/77bb9edd-d18b-43b8-bc40-cebb018f936c_calibrate_898798_33.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/77bb9edd-d18b-43b8-bc40-cebb018f936c_calibrate_898798_33.stderr parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/7d034d54-acbf-48a5-bfd8-81626a5d0dd5_characterizeImage_898798_33.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/7d034d54-acbf-48a5-bfd8-81626a5d0dd5_characterizeImage_898798_33.stderr parsl.dataflow.dflow INFO: Task 388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2dfcfff2270fa7d929a30db7700cad80 with result from task 388 parsl.dataflow.dflow DEBUG: Task 388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 389 has memoization hash 37c857d99abbe016a9161bd1cb84d4c8 parsl.dataflow.memoization INFO: Task 389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d662340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 851 parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/129a4036-131a-4c10-bb6b-d32c33380bb3_calibrate_898694_38.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/129a4036-131a-4c10-bb6b-d32c33380bb3_calibrate_898694_38.stderr parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/27010e38-6ed4-4b48-ad4f-ccb5567b8c9c_characterizeImage_898694_38.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/27010e38-6ed4-4b48-ad4f-ccb5567b8c9c_characterizeImage_898694_38.stderr parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a12f3c08fe4c8cbf0c07c30219508502 with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash 1f45774fa008e93025764dd47e2d2cfa parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 852 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/75816b91-9de4-4ef6-afcd-445b7939a547_calibrate_898694_15.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/75816b91-9de4-4ef6-afcd-445b7939a547_calibrate_898694_15.stderr parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/1f323fe0-ea9a-45aa-8fa5-57485168192d_characterizeImage_898694_15.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/1f323fe0-ea9a-45aa-8fa5-57485168192d_characterizeImage_898694_15.stderr parsl.dataflow.dflow INFO: Task 745 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d49705fd931b6259cf06ba5368575ec6 with result from task 745 parsl.dataflow.dflow DEBUG: Task 745 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 746 has memoization hash 096d1ac1a5e400abfa77dfd1d98c9317 parsl.dataflow.memoization INFO: Task 746 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 746 try 0 launched on executor multi with executor id 853 parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/8cb66809-92a3-4a93-8bd1-52bf5934689d_calibrate_898798_47.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/8cb66809-92a3-4a93-8bd1-52bf5934689d_calibrate_898798_47.stderr parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/2c42c1e2-3d10-4c68-8e76-edd81a807591_characterizeImage_898798_47.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/2c42c1e2-3d10-4c68-8e76-edd81a807591_characterizeImage_898798_47.stderr parsl.dataflow.dflow INFO: Task 565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4299c550db0ff2d67e126a33e2a6232 with result from task 565 parsl.dataflow.dflow DEBUG: Task 565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 566 has memoization hash ea64ee82c6d7a87f049fb90ecac0d0ff parsl.dataflow.memoization INFO: Task 566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4ddda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 566 try 0 launched on executor multi with executor id 854 parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/c8f64e27-16de-4a86-b664-8a64081be740_calibrate_898696_17.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/c8f64e27-16de-4a86-b664-8a64081be740_calibrate_898696_17.stderr parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/73021dd1-29da-4355-ac8d-213bdb1483e2_characterizeImage_898696_17.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/73021dd1-29da-4355-ac8d-213bdb1483e2_characterizeImage_898696_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:22 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96900eab0cf83c9a97458a39ed0a9610 with result from task 547 parsl.dataflow.dflow DEBUG: Task 547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 548 has memoization hash 6bdf42b7cf1c64982018beafe7976e83 parsl.dataflow.memoization INFO: Task 548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 548 try 0 launched on executor multi with executor id 855 parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/d888f731-a78b-4400-b9bd-104a3dd2bf94_calibrate_898696_37.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/d888f731-a78b-4400-b9bd-104a3dd2bf94_calibrate_898696_37.stderr parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/31a6430d-764c-430e-a4c5-2beadf7afd57_characterizeImage_898696_37.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/31a6430d-764c-430e-a4c5-2beadf7afd57_characterizeImage_898696_37.stderr parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8ca691e9ad71f1feee3946b6fd92372 with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash 6530e4c8508cbaf9358727c2eac7fcc8 parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d609940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 856 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/c89110c6-4ebc-4113-8f35-3e4027170324_calibrate_898798_23.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/c89110c6-4ebc-4113-8f35-3e4027170324_calibrate_898798_23.stderr parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/2f9bcff7-43eb-47ad-a327-cbc316d4b671_characterizeImage_898798_23.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/2f9bcff7-43eb-47ad-a327-cbc316d4b671_characterizeImage_898798_23.stderr parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5000eeee80af68bf6cf9a1472c211ee with result from task 433 parsl.dataflow.dflow DEBUG: Task 433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 434 has memoization hash 7edb1b9fa5b27393e5b2852c0c5dcdf2 parsl.dataflow.memoization INFO: Task 434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c481440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 857 parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/de81f830-7059-48f3-9753-b82c9e9b9ed8_calibrate_898800_53.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/de81f830-7059-48f3-9753-b82c9e9b9ed8_calibrate_898800_53.stderr parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/469ce836-8c34-47f8-afc5-72c34a62505c_characterizeImage_898800_53.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/469ce836-8c34-47f8-afc5-72c34a62505c_characterizeImage_898800_53.stderr parsl.dataflow.dflow INFO: Task 706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71f0cac51c0a7d73b9cc7325a7337bc4 with result from task 706 parsl.dataflow.dflow DEBUG: Task 706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 707 has memoization hash efc27564e9a83610ca70659571353910 parsl.dataflow.memoization INFO: Task 707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c377380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 707 try 0 launched on executor multi with executor id 858 parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/670eb3c8-44bb-4ec2-9e3e-72a7488abaae_calibrate_898696_47.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/670eb3c8-44bb-4ec2-9e3e-72a7488abaae_calibrate_898696_47.stderr parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/fd62df96-855b-4c72-82b0-8f1e161cb8e2_characterizeImage_898696_47.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/fd62df96-855b-4c72-82b0-8f1e161cb8e2_characterizeImage_898696_47.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93111b41600e07a57ded45baedcd9620 with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 4375ee0f2cf507ea3963886d919513a8 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 859 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/9e15c895-4628-44bb-9433-c8814624c1a9_calibrate_898694_36.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/9e15c895-4628-44bb-9433-c8814624c1a9_calibrate_898694_36.stderr parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/595267a1-4cc6-4ac4-bb51-5a7a935e4ff4_characterizeImage_898694_36.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/595267a1-4cc6-4ac4-bb51-5a7a935e4ff4_characterizeImage_898694_36.stderr parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 424136afcaa41ddfea55e9669cf9c63d with result from task 307 parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash 16d81a6c22afdbc61bef9a7351cdef93 parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 860 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/402a1bdb-108a-406b-b988-c2ebcecf93d8_calibrate_898696_26.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/402a1bdb-108a-406b-b988-c2ebcecf93d8_calibrate_898696_26.stderr parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/ac40a5bd-c079-4198-acde-70e297b5af3a_characterizeImage_898696_26.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/ac40a5bd-c079-4198-acde-70e297b5af3a_characterizeImage_898696_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:23 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 1003 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a6c16b5c810ba0e2a37790a972d5d82 with result from task 1003 parsl.dataflow.dflow DEBUG: Task 1003 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1004 has memoization hash 1a1b6f50640eae038defe18fd6fa5cf0 parsl.dataflow.memoization INFO: Task 1004 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2daa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1004 try 0 launched on executor multi with executor id 861 parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/80f481bd-aeb1-4c15-aad4-b47936b66dea_calibrate_898799_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/80f481bd-aeb1-4c15-aad4-b47936b66dea_calibrate_898799_53.stderr parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/bf09297e-f6f0-474a-921f-aecf3b613097_characterizeImage_898799_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/bf09297e-f6f0-474a-921f-aecf3b613097_characterizeImage_898799_53.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d7933ed13c8c642d3d69e5b0d16f557 with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 849755fae16d165a7f1cc1477c3c3cd9 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 862 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/d48db75e-100e-4636-a590-5ca468cf221a_calibrate_898799_46.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/d48db75e-100e-4636-a590-5ca468cf221a_calibrate_898799_46.stderr parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/93b7d5c3-a04f-4f3a-9d8d-435d24b80257_characterizeImage_898799_46.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/93b7d5c3-a04f-4f3a-9d8d-435d24b80257_characterizeImage_898799_46.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 664 try 0 failed parsl.dataflow.dflow INFO: Task 664 marked for retry parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/0d8e4103-985c-4101-9969-cc0402388124_characterizeImage_898694_2.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/0d8e4103-985c-4101-9969-cc0402388124_characterizeImage_898694_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 664 has memoization hash b39895c145d9c2f38f6a6458ac769dbb parsl.dataflow.memoization INFO: Task 664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c374040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 664 try 1 launched on executor multi with executor id 863 parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/0d8e4103-985c-4101-9969-cc0402388124_characterizeImage_898694_2.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/0d8e4103-985c-4101-9969-cc0402388124_characterizeImage_898694_2.stderr parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea2e8e492d2c7029be923f548281db20 with result from task 403 parsl.dataflow.dflow DEBUG: Task 403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 404 has memoization hash e7345f2b2eea36b06aa3d3d6df5fdcb9 parsl.dataflow.memoization INFO: Task 404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d663600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 864 parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/b07c55c9-c8b4-445c-88e8-4824be98fa2f_calibrate_898800_62.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/b07c55c9-c8b4-445c-88e8-4824be98fa2f_calibrate_898800_62.stderr parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/af3964b9-6412-45b2-b6aa-8529883dd89b_characterizeImage_898800_62.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/af3964b9-6412-45b2-b6aa-8529883dd89b_characterizeImage_898800_62.stderr parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd8f97d421afc5c28d1d8c0b519784d3 with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash db7179dfb22c8b75c302ef687ebcf361 parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d608fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 865 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/9ea68e1d-b4e6-409a-a172-5486a0efce9a_calibrate_898694_41.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/9ea68e1d-b4e6-409a-a172-5486a0efce9a_calibrate_898694_41.stderr parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/bc488d99-7876-42b4-bc90-e5b8c6282f5a_characterizeImage_898694_41.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/bc488d99-7876-42b4-bc90-e5b8c6282f5a_characterizeImage_898694_41.stderr parsl.dataflow.dflow INFO: Task 742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 201b534dd50008d065a3f9e127f1262b with result from task 742 parsl.dataflow.dflow DEBUG: Task 742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 743 has memoization hash ec1ae208f2ef361e5a2a7a1ca2102083 parsl.dataflow.memoization INFO: Task 743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 743 try 0 launched on executor multi with executor id 866 parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/bfc95f0f-e3a1-4524-9dbc-748b03680c63_calibrate_898694_24.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/bfc95f0f-e3a1-4524-9dbc-748b03680c63_calibrate_898694_24.stderr parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/0e90c368-ee12-4940-a195-be79d1145d14_characterizeImage_898694_24.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/0e90c368-ee12-4940-a195-be79d1145d14_characterizeImage_898694_24.stderr parsl.dataflow.dflow INFO: Task 616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38222c296165b6da0fe0b1f493077f40 with result from task 616 parsl.dataflow.dflow DEBUG: Task 616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 617 has memoization hash 48914a51d7f76b8228ef0f23adfd70c8 parsl.dataflow.memoization INFO: Task 617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c511580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 617 try 0 launched on executor multi with executor id 867 parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/dfa40ab7-a172-4536-9004-5461859ff661_calibrate_898695_22.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/dfa40ab7-a172-4536-9004-5461859ff661_calibrate_898695_22.stderr parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/4b7c1047-7cd8-456e-8371-2b489c534f3e_characterizeImage_898695_22.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/4b7c1047-7cd8-456e-8371-2b489c534f3e_characterizeImage_898695_22.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0351226e6d5e7e58e1537bffb1bd4673 with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 4d6bbc9bd39ea774535d51b005649bdc parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cc900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 868 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/9478d13b-30c4-4be1-a5f7-f81fd1dd2aad_calibrate_898696_13.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/9478d13b-30c4-4be1-a5f7-f81fd1dd2aad_calibrate_898696_13.stderr parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/79e0dd7c-83a9-4498-83af-7520b03515a2_characterizeImage_898696_13.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/79e0dd7c-83a9-4498-83af-7520b03515a2_characterizeImage_898696_13.stderr parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd169fcb73b46fb70e0cf782149ff958 with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 69cdb759d815a58732504629c5756ced parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5eed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 869 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/269c8fe2-fbf0-4a3c-8939-728692b548f4_calibrate_898694_60.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/269c8fe2-fbf0-4a3c-8939-728692b548f4_calibrate_898694_60.stderr parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/dd3d308e-4d92-428f-b494-4693af80b880_characterizeImage_898694_60.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/dd3d308e-4d92-428f-b494-4693af80b880_characterizeImage_898694_60.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 391 try 0 failed parsl.dataflow.dflow INFO: Task 391 marked for retry parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/3143bd25-3fe9-4373-ad0c-e4d787503f78_characterizeImage_898695_2.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/3143bd25-3fe9-4373-ad0c-e4d787503f78_characterizeImage_898695_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash 803724b20eb447b5fee3f20f596264d3 parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d662840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 1 launched on executor multi with executor id 870 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/3143bd25-3fe9-4373-ad0c-e4d787503f78_characterizeImage_898695_2.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/3143bd25-3fe9-4373-ad0c-e4d787503f78_characterizeImage_898695_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:24 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 850 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a75c23c1d66ab4d9ed827e0ebf204ab1 with result from task 850 parsl.dataflow.dflow DEBUG: Task 850 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 851 has memoization hash c3ac6c8c5f1c51ae8d0634f21a408df2 parsl.dataflow.memoization INFO: Task 851 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c440a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 851 try 0 launched on executor multi with executor id 871 parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/70b3ab77-65b4-413a-bdfa-4e3c65325dc7_calibrate_898799_56.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/70b3ab77-65b4-413a-bdfa-4e3c65325dc7_calibrate_898799_56.stderr parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/b9945013-2e60-4ee1-951f-164928e8e983_characterizeImage_898799_56.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/b9945013-2e60-4ee1-951f-164928e8e983_characterizeImage_898799_56.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 156af6c974e7f50c0d95883c66f06b5b with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 1d4592c5b24f9fa9c3f69558d25fff31 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d585da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 872 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/b18dbaa1-f478-4fda-9ea3-61679c5ab540_calibrate_898695_45.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/b18dbaa1-f478-4fda-9ea3-61679c5ab540_calibrate_898695_45.stderr parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/61b94d0e-4e1b-4091-ba62-b77cecda938d_characterizeImage_898695_45.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/61b94d0e-4e1b-4091-ba62-b77cecda938d_characterizeImage_898695_45.stderr parsl.dataflow.dflow INFO: Task 658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02135cf0533727b74fc36577ed1a6fec with result from task 658 parsl.dataflow.dflow DEBUG: Task 658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 659 has memoization hash 7ce4651bd3cf31b6bf054c7a7016a678 parsl.dataflow.memoization INFO: Task 659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c376c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 659 try 0 launched on executor multi with executor id 873 parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/4777370d-acb7-45e3-8d29-fec3e2891b56_calibrate_898696_22.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/4777370d-acb7-45e3-8d29-fec3e2891b56_calibrate_898696_22.stderr parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/e6ae0ab7-345c-40ca-ac11-9d65cec3c4a7_characterizeImage_898696_22.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/e6ae0ab7-345c-40ca-ac11-9d65cec3c4a7_characterizeImage_898696_22.stderr parsl.dataflow.dflow INFO: Task 1066 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ec77fd71990744c36a3d147ec99a4e0 with result from task 1066 parsl.dataflow.dflow DEBUG: Task 1066 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1067 has memoization hash 5b37a3dc89a7eb1c0d6ad59523dcbec7 parsl.dataflow.memoization INFO: Task 1067 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c328fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1067 try 0 launched on executor multi with executor id 874 parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/9e8d8f81-dab4-4261-a1d3-8355cf83d399_calibrate_898798_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/9e8d8f81-dab4-4261-a1d3-8355cf83d399_calibrate_898798_37.stderr parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/550cc592-e72e-4cda-955a-ef527abbb87b_characterizeImage_898798_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/550cc592-e72e-4cda-955a-ef527abbb87b_characterizeImage_898798_37.stderr parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0efeed125ed8a35b317a446da53f525 with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash 813d931c046c9836b74a99f4f389670b parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 875 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/377242ec-5005-4e58-82c4-26bd207a9818_calibrate_898694_14.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/377242ec-5005-4e58-82c4-26bd207a9818_calibrate_898694_14.stderr parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/ed61939e-7bf4-4605-8285-b41df14ef57c_characterizeImage_898694_14.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/ed61939e-7bf4-4605-8285-b41df14ef57c_characterizeImage_898694_14.stderr parsl.dataflow.dflow INFO: Task 940 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 563a0e5f330b82c63dbe453d54e75540 with result from task 940 parsl.dataflow.dflow DEBUG: Task 940 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 941 has memoization hash b2cc7df5b82607c08731b0ca8a98c675 parsl.dataflow.memoization INFO: Task 941 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c291f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 941 try 0 launched on executor multi with executor id 876 parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/7f8e6443-4c15-42b3-8ddd-ba570570a884_calibrate_898798_21.stdout parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/7f8e6443-4c15-42b3-8ddd-ba570570a884_calibrate_898798_21.stderr parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ff0948ec-257e-48f5-b058-e0a54de01984_characterizeImage_898798_21.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ff0948ec-257e-48f5-b058-e0a54de01984_characterizeImage_898798_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:26 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 985 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e5d37e54b81bfda4aef3069745772f6 with result from task 985 parsl.dataflow.dflow DEBUG: Task 985 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 986 has memoization hash d7c0232eedc1803bf91eae0d4a049fa6 parsl.dataflow.memoization INFO: Task 986 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 986 try 0 launched on executor multi with executor id 877 parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/a37af44c-7028-450a-99f4-d2d587bef4b7_calibrate_898798_12.stdout parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/a37af44c-7028-450a-99f4-d2d587bef4b7_calibrate_898798_12.stderr parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/d536fa26-e9ad-490a-962e-9cd0623ecf87_characterizeImage_898798_12.stdout parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/d536fa26-e9ad-490a-962e-9cd0623ecf87_characterizeImage_898798_12.stderr parsl.dataflow.dflow INFO: Task 859 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2daa96f58b5cbdeb4890ce2a076319c3 with result from task 859 parsl.dataflow.dflow DEBUG: Task 859 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 860 has memoization hash c085d3eafd673eae115b054892667dda parsl.dataflow.memoization INFO: Task 860 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4413a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 860 try 0 launched on executor multi with executor id 878 parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/076ce221-76fe-4523-8b31-0d6f6affe420_calibrate_898695_30.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/076ce221-76fe-4523-8b31-0d6f6affe420_calibrate_898695_30.stderr parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/6fb44520-31ca-4603-8658-7f010273562a_characterizeImage_898695_30.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/6fb44520-31ca-4603-8658-7f010273562a_characterizeImage_898695_30.stderr parsl.dataflow.dflow INFO: Task 1009 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cdec4012c9852d23b211abb235d1efc with result from task 1009 parsl.dataflow.dflow DEBUG: Task 1009 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1010 has memoization hash c4ed22444bc18eb2dd2cfa0332bc067e parsl.dataflow.memoization INFO: Task 1010 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2db060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1010 try 0 launched on executor multi with executor id 879 parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/b454d900-6de6-4f09-9642-5aa3608aee1b_calibrate_898798_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/b454d900-6de6-4f09-9642-5aa3608aee1b_calibrate_898798_8.stderr parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/3aab0bda-1981-4fb6-ae9e-397916d8dd80_characterizeImage_898798_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/3aab0bda-1981-4fb6-ae9e-397916d8dd80_characterizeImage_898798_8.stderr parsl.dataflow.dflow INFO: Task 874 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee8993e920e0bb103fc95bb9b60c287b with result from task 874 parsl.dataflow.dflow DEBUG: Task 874 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 875 has memoization hash 2a1b0efb8f1e8180a54734b4399d870c parsl.dataflow.memoization INFO: Task 875 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c442340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 875 try 0 launched on executor multi with executor id 880 parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/f2fcbf99-de8d-4ac9-8e60-d849391ebd14_calibrate_898694_37.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/f2fcbf99-de8d-4ac9-8e60-d849391ebd14_calibrate_898694_37.stderr parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/c8858d83-ca4a-4ea3-875c-de04eccf789c_characterizeImage_898694_37.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/c8858d83-ca4a-4ea3-875c-de04eccf789c_characterizeImage_898694_37.stderr parsl.dataflow.dflow INFO: Task 379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6788856663694cdf56309b93a33c4541 with result from task 379 parsl.dataflow.dflow DEBUG: Task 379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 380 has memoization hash f32ed1802853a727f49734310071b183 parsl.dataflow.memoization INFO: Task 380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6619e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 881 parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/3e9d780e-bcb7-4a31-b311-3da39b8b2275_calibrate_898798_38.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/3e9d780e-bcb7-4a31-b311-3da39b8b2275_calibrate_898798_38.stderr parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/2fd47b84-d7f3-402d-a83c-fe069932c762_characterizeImage_898798_38.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/2fd47b84-d7f3-402d-a83c-fe069932c762_characterizeImage_898798_38.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b53183c7d188d64aa53806008a5ddb0 with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 1d7d9c95c5c00ef95f2123a8f4702f53 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d586340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 882 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/78f801df-8bad-474a-83c9-3519913117e2_calibrate_898798_4.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/78f801df-8bad-474a-83c9-3519913117e2_calibrate_898798_4.stderr parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/a49e8108-af1c-4307-8cf1-08e5e3451426_characterizeImage_898798_4.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/a49e8108-af1c-4307-8cf1-08e5e3451426_characterizeImage_898798_4.stderr parsl.dataflow.dflow INFO: Task 856 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d4a03d116e8e5bb24b6d4e1ab8099db with result from task 856 parsl.dataflow.dflow DEBUG: Task 856 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 857 has memoization hash 49a23227f1c192ef6d12a2b450a25e26 parsl.dataflow.memoization INFO: Task 857 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c441080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 857 try 0 launched on executor multi with executor id 883 parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/0032e136-30f0-4506-a459-f2ec1fe3d711_calibrate_898694_51.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/0032e136-30f0-4506-a459-f2ec1fe3d711_calibrate_898694_51.stderr parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/48405b23-b761-43cc-a39a-298193b8114a_characterizeImage_898694_51.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/48405b23-b761-43cc-a39a-298193b8114a_characterizeImage_898694_51.stderr parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0318b0119f1f8bbe0839f7f4e78ead2 with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash 6a87d311c7482a562ade51501fed9a0e parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d609620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 884 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/bb9391b3-d020-4abd-889b-33252bd79234_calibrate_898799_32.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/bb9391b3-d020-4abd-889b-33252bd79234_calibrate_898799_32.stderr parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/6abe174d-4883-4a2f-832e-1ba138e39c3e_characterizeImage_898799_32.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/6abe174d-4883-4a2f-832e-1ba138e39c3e_characterizeImage_898799_32.stderr parsl.dataflow.dflow INFO: Task 490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf65f16682d1373f3e771f0570906e9f with result from task 490 parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:27 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow DEBUG: Task 490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 491 has memoization hash ed9d1408999177723e0f5283c03acc42 parsl.dataflow.memoization INFO: Task 491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c483920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 491 try 0 launched on executor multi with executor id 885 parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/75f29b4a-3a0e-4785-b51f-87f276a46cce_calibrate_898798_62.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/75f29b4a-3a0e-4785-b51f-87f276a46cce_calibrate_898798_62.stderr parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ce6b58e3-4ded-415f-9235-bdd55175c8c8_characterizeImage_898798_62.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ce6b58e3-4ded-415f-9235-bdd55175c8c8_characterizeImage_898798_62.stderr parsl.dataflow.dflow INFO: Task 736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5a5e9fcd721475d2e071614a95870c0 with result from task 736 parsl.dataflow.dflow DEBUG: Task 736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 737 has memoization hash 68b3fd120edaf3ac2efdf05f8ddaee9e parsl.dataflow.memoization INFO: Task 737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 737 try 0 launched on executor multi with executor id 886 parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/c26a49d5-9855-437b-9d11-d3f4685fa3dc_calibrate_898696_46.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/c26a49d5-9855-437b-9d11-d3f4685fa3dc_calibrate_898696_46.stderr parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/859c6268-5814-454b-b921-30f548078b3a_characterizeImage_898696_46.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/859c6268-5814-454b-b921-30f548078b3a_characterizeImage_898696_46.stderr parsl.dataflow.dflow INFO: Task 409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31768357e251dce6afe3aa277f54e222 with result from task 409 parsl.dataflow.dflow DEBUG: Task 409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 410 has memoization hash 6bacbfdf224a4a0d6193d3af9ac502fa parsl.dataflow.memoization INFO: Task 410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d663ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 410 try 0 launched on executor multi with executor id 887 parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/12ca89e5-9b0b-4eb9-ad8c-617dfaddab06_calibrate_898799_7.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/12ca89e5-9b0b-4eb9-ad8c-617dfaddab06_calibrate_898799_7.stderr parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/54f83227-aa48-4a4c-8ea8-c24c53810802_characterizeImage_898799_7.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/54f83227-aa48-4a4c-8ea8-c24c53810802_characterizeImage_898799_7.stderr parsl.dataflow.dflow INFO: Task 964 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5aec3021e41463df3c5593077b22d8e4 with result from task 964 parsl.dataflow.dflow DEBUG: Task 964 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 965 has memoization hash 2abef293a75b13a457be0049b4eb5977 parsl.dataflow.memoization INFO: Task 965 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 965 try 0 launched on executor multi with executor id 888 parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/b728b248-858e-4358-be28-1829ce4ca35b_calibrate_898799_51.stdout parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/b728b248-858e-4358-be28-1829ce4ca35b_calibrate_898799_51.stderr parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/3793ed74-8e63-47a9-9c1d-655dd3a24540_characterizeImage_898799_51.stdout parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/3793ed74-8e63-47a9-9c1d-655dd3a24540_characterizeImage_898799_51.stderr parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1fb2b13946de12363647c40ba5f21fc with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash bc8a411efb8421bb3a4d4b472c7ecd47 parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ef6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 889 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/8e28a9b9-03a1-48a3-888d-16b9f0042a28_calibrate_898798_10.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/8e28a9b9-03a1-48a3-888d-16b9f0042a28_calibrate_898798_10.stderr parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/b20aa47c-1b01-4f0d-b4dd-efdddaf10838_characterizeImage_898798_10.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/b20aa47c-1b01-4f0d-b4dd-efdddaf10838_characterizeImage_898798_10.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5cd93403189d5ee5111193205696724 with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 3ed2c7960d80d130b6c6d53a793dafdf parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ec360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 890 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/a031f834-08a6-499d-803d-6419e50b7fed_calibrate_898694_29.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/a031f834-08a6-499d-803d-6419e50b7fed_calibrate_898694_29.stderr parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/ff851375-ed26-428a-a051-c1d67a1b01fe_characterizeImage_898694_29.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/ff851375-ed26-428a-a051-c1d67a1b01fe_characterizeImage_898694_29.stderr parsl.dataflow.dflow INFO: Task 766 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74fbd11f427a0d15ff2e4a8cb33a8230 with result from task 766 parsl.dataflow.dflow DEBUG: Task 766 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 767 has memoization hash b240053786cc7ba4f61c5f1331c125c0 parsl.dataflow.memoization INFO: Task 767 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 767 try 0 launched on executor multi with executor id 891 parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/f57f0dc2-d049-4398-b58a-cb4a6e16a651_calibrate_898695_32.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/f57f0dc2-d049-4398-b58a-cb4a6e16a651_calibrate_898695_32.stderr parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/d923957f-e169-470c-92ea-e9457d194cfc_characterizeImage_898695_32.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/d923957f-e169-470c-92ea-e9457d194cfc_characterizeImage_898695_32.stderr parsl.dataflow.dflow INFO: Task 958 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1756d9c7b108818adade0d2723e32b76 with result from task 958 parsl.dataflow.dflow DEBUG: Task 958 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 959 has memoization hash 3a69308867f2396776a442fc4746025f parsl.dataflow.memoization INFO: Task 959 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2931a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 959 try 0 launched on executor multi with executor id 892 parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/2d5aa007-4806-4261-8a14-8e2434ddde20_calibrate_898798_51.stdout parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/2d5aa007-4806-4261-8a14-8e2434ddde20_calibrate_898798_51.stderr parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/0c6ef654-93be-490f-9d94-d6e3faed54a7_characterizeImage_898798_51.stdout parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/0c6ef654-93be-490f-9d94-d6e3faed54a7_characterizeImage_898798_51.stderr parsl.dataflow.dflow INFO: Task 934 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f0520a6ae3d96d6759f77b5f190e7b4 with result from task 934 parsl.dataflow.dflow DEBUG: Task 934 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 935 has memoization hash a6eac639a95c7ad441670a94d81b14a7 parsl.dataflow.memoization INFO: Task 935 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2918a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 935 try 0 launched on executor multi with executor id 893 parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/1417e7f6-7eaa-4d65-9775-90c2c979913b_calibrate_898696_33.stdout parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/1417e7f6-7eaa-4d65-9775-90c2c979913b_calibrate_898696_33.stderr parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/0dab2b7d-1a60-4de5-afd4-334236346842_characterizeImage_898696_33.stdout parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/0dab2b7d-1a60-4de5-afd4-334236346842_characterizeImage_898696_33.stderr parsl.dataflow.dflow INFO: Task 769 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edc57ca699368f7750566381a2edec44 with result from task 769 parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:28 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow DEBUG: Task 769 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 770 has memoization hash 6632e7a1fa5c99af4945d3f09f0964b8 parsl.dataflow.memoization INFO: Task 770 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 770 try 0 launched on executor multi with executor id 894 parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/4ab11c6a-a28f-4e2d-ac12-cb21c0241fd9_calibrate_898800_10.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/4ab11c6a-a28f-4e2d-ac12-cb21c0241fd9_calibrate_898800_10.stderr parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/d1c24002-b728-418a-8c26-05f05da239ec_characterizeImage_898800_10.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/d1c24002-b728-418a-8c26-05f05da239ec_characterizeImage_898800_10.stderr parsl.dataflow.dflow INFO: Task 487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcc595aa24df6fa0a580b03d8d9048be with result from task 487 parsl.dataflow.dflow DEBUG: Task 487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 488 has memoization hash 50f69a7a97754aad8e679c65a0b60a89 parsl.dataflow.memoization INFO: Task 488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c483600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 488 try 0 launched on executor multi with executor id 895 parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/91aac08c-c5f3-4ec6-aad5-f40007aa5700_calibrate_898694_27.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/91aac08c-c5f3-4ec6-aad5-f40007aa5700_calibrate_898694_27.stderr parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/6917b83a-7673-43f6-9ccf-bb14004f3dee_characterizeImage_898694_27.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/6917b83a-7673-43f6-9ccf-bb14004f3dee_characterizeImage_898694_27.stderr parsl.dataflow.dflow INFO: Task 904 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c33284c3be1e00cc9bff47ea03e94e9b with result from task 904 parsl.dataflow.dflow DEBUG: Task 904 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 905 has memoization hash f54cf6470881686cb45f3c6f767e95a0 parsl.dataflow.memoization INFO: Task 905 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2902c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 905 try 0 launched on executor multi with executor id 896 parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/44c9aac1-e91d-4eb3-b49f-0bc0f4d90f6f_calibrate_898694_59.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/44c9aac1-e91d-4eb3-b49f-0bc0f4d90f6f_calibrate_898694_59.stderr parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/565605a4-c6bf-49c2-af54-e90a342b2fe5_characterizeImage_898694_59.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/565605a4-c6bf-49c2-af54-e90a342b2fe5_characterizeImage_898694_59.stderr parsl.dataflow.dflow INFO: Task 1096 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc60ab7e53278bc689c38ffea5849e69 with result from task 1096 parsl.dataflow.dflow DEBUG: Task 1096 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1097 has memoization hash 0cd46b9d2ae719cf1b0165b80d74be16 parsl.dataflow.memoization INFO: Task 1097 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c350b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1097 try 0 launched on executor multi with executor id 897 parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/3ff5c90d-cecc-454c-813b-e659c8c8bf13_calibrate_898798_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/3ff5c90d-cecc-454c-813b-e659c8c8bf13_calibrate_898798_40.stderr parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/977970fa-9fd4-42ee-977b-81128df035ac_characterizeImage_898798_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/977970fa-9fd4-42ee-977b-81128df035ac_characterizeImage_898798_40.stderr parsl.dataflow.dflow INFO: Task 511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92ae2f671eb39cd112f41db07d987799 with result from task 511 parsl.dataflow.dflow DEBUG: Task 511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 512 has memoization hash 6c9784ca3982c966bdf1306df3e199f7 parsl.dataflow.memoization INFO: Task 512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 512 try 0 launched on executor multi with executor id 898 parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/04b032ee-0b81-4b2a-bea8-5c692d74a988_calibrate_898799_11.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/04b032ee-0b81-4b2a-bea8-5c692d74a988_calibrate_898799_11.stderr parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/e7b9bf64-2407-4a77-8bf6-d09ef063c948_characterizeImage_898799_11.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/e7b9bf64-2407-4a77-8bf6-d09ef063c948_characterizeImage_898799_11.stderr parsl.dataflow.dflow INFO: Task 976 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b18258c1503b9bfe586b1ebcd4aa4af with result from task 976 parsl.dataflow.dflow DEBUG: Task 976 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 977 has memoization hash b3b8606a258fdcb0abd19d8f1a6187d6 parsl.dataflow.memoization INFO: Task 977 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 977 try 0 launched on executor multi with executor id 899 parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/e3370a5b-cd5a-42f2-b070-7dfb3c3c6d70_calibrate_898799_57.stdout parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/e3370a5b-cd5a-42f2-b070-7dfb3c3c6d70_calibrate_898799_57.stderr parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/4a724df4-da41-432e-ad2d-7c5b3d81f45f_characterizeImage_898799_57.stdout parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/4a724df4-da41-432e-ad2d-7c5b3d81f45f_characterizeImage_898799_57.stderr parsl.dataflow.dflow INFO: Task 718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf261c7f1e6529a20438313511264a18 with result from task 718 parsl.dataflow.dflow DEBUG: Task 718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 719 has memoization hash 50bc805d3a349b6ce51cfde3a4fe26de parsl.dataflow.memoization INFO: Task 719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 719 try 0 launched on executor multi with executor id 900 parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/f2087b2c-2eca-43b6-b779-4e1af7a975e4_calibrate_898798_39.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/f2087b2c-2eca-43b6-b779-4e1af7a975e4_calibrate_898798_39.stderr parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/eb4fed2f-82dd-4a37-9508-fb1a3e8d479e_characterizeImage_898798_39.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/eb4fed2f-82dd-4a37-9508-fb1a3e8d479e_characterizeImage_898798_39.stderr parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05e2d1f0cd130a1c42243e02d20de459 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 92edf577eb80fb925f662554cef03695 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cfb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 901 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/293243b8-d4c4-4189-bb32-49dd7dcea659_calibrate_898696_52.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/293243b8-d4c4-4189-bb32-49dd7dcea659_calibrate_898696_52.stderr parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/6ca1e530-13fa-4c33-833c-6568428a18ce_characterizeImage_898696_52.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/6ca1e530-13fa-4c33-833c-6568428a18ce_characterizeImage_898696_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:30 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c51004598d45827d1a4f9a387e039a1 with result from task 472 parsl.dataflow.dflow DEBUG: Task 472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 473 has memoization hash 9362e675d52d92d9c0b9bb113d89e7c9 parsl.dataflow.memoization INFO: Task 473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c482520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 473 try 0 launched on executor multi with executor id 902 parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/5f289302-85d2-4532-af71-d7a3f9209595_calibrate_898695_14.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/5f289302-85d2-4532-af71-d7a3f9209595_calibrate_898695_14.stderr parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/f9015afc-4f8f-4f6a-9e0d-944dd4b89f5b_characterizeImage_898695_14.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/f9015afc-4f8f-4f6a-9e0d-944dd4b89f5b_characterizeImage_898695_14.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4bf2154eb7b8e3ce1f2ce6b5b174ac3 with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash aee323f091533fe209bb59d64f7841e5 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ce840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 903 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/f0dc286a-344c-4c43-90b4-aebfa5ad6f35_calibrate_898694_44.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/f0dc286a-344c-4c43-90b4-aebfa5ad6f35_calibrate_898694_44.stderr parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/05e04231-940f-4e68-bcf5-507976033d30_characterizeImage_898694_44.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/05e04231-940f-4e68-bcf5-507976033d30_characterizeImage_898694_44.stderr parsl.dataflow.dflow INFO: Task 1087 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32ccebbcb10c5dbe283275dbbe2e9165 with result from task 1087 parsl.dataflow.dflow DEBUG: Task 1087 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1088 has memoization hash b3e31c999c23bad8c4a6abbd284126a9 parsl.dataflow.memoization INFO: Task 1088 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c350220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1088 try 0 launched on executor multi with executor id 904 parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/bd8d2f44-8b63-4eff-a2e2-2523baadf0cc_calibrate_898800_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/bd8d2f44-8b63-4eff-a2e2-2523baadf0cc_calibrate_898800_16.stderr parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/e8e7027e-0930-465c-b516-ae76c660cd30_characterizeImage_898800_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/e8e7027e-0930-465c-b516-ae76c660cd30_characterizeImage_898800_16.stderr parsl.dataflow.dflow INFO: Task 841 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 677a066dbf70efb0c278b0e9a8e48509 with result from task 841 parsl.dataflow.dflow DEBUG: Task 841 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 842 has memoization hash 864ae8a5c32dc3f51f3ac622e2e42ef8 parsl.dataflow.memoization INFO: Task 842 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4400e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 842 try 0 launched on executor multi with executor id 905 parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/957d4535-421d-4fc0-bf2b-ae2507f4bb89_calibrate_898696_14.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/957d4535-421d-4fc0-bf2b-ae2507f4bb89_calibrate_898696_14.stderr parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/6fa874dd-14e1-4366-a53a-b2f4652e25ab_characterizeImage_898696_14.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/6fa874dd-14e1-4366-a53a-b2f4652e25ab_characterizeImage_898696_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:31 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aef361fbde57b26027fd447201f6a62e with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash f9ab423a8ea28930aada304b92b51bd2 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 906 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/46da9d85-c5bf-4321-9280-8ef233ccdccc_calibrate_898695_44.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/46da9d85-c5bf-4321-9280-8ef233ccdccc_calibrate_898695_44.stderr parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/9f6776cb-79d7-4059-a952-a6e36d43067c_characterizeImage_898695_44.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/9f6776cb-79d7-4059-a952-a6e36d43067c_characterizeImage_898695_44.stderr parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bdc683351e9ff8b51ca33427b1f39f2 with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash 522af67f60b98c0880b2b6fe504ab6da parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 907 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/57b37766-3d7e-4bba-af71-f66c5fa17154_calibrate_898799_5.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/57b37766-3d7e-4bba-af71-f66c5fa17154_calibrate_898799_5.stderr parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/ce7e3124-805e-4a57-851d-44abc875c31e_characterizeImage_898799_5.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/ce7e3124-805e-4a57-851d-44abc875c31e_characterizeImage_898799_5.stderr parsl.dataflow.dflow INFO: Task 1045 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3f92e664aa2baa96cb9104c0694f336 with result from task 1045 parsl.dataflow.dflow DEBUG: Task 1045 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1046 has memoization hash 0173d496d95ddd7738912cacf3f2a44d parsl.dataflow.memoization INFO: Task 1046 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1046 try 0 launched on executor multi with executor id 908 parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/a29b26be-ff63-4599-b333-98b5af42916c_calibrate_898800_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/a29b26be-ff63-4599-b333-98b5af42916c_calibrate_898800_8.stderr parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/b6ed734c-f6ca-449f-a517-ab9935c2766b_characterizeImage_898800_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/b6ed734c-f6ca-449f-a517-ab9935c2766b_characterizeImage_898800_8.stderr parsl.dataflow.dflow INFO: Task 646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92435ff8ae64041088cfe8442f47f402 with result from task 646 parsl.dataflow.dflow DEBUG: Task 646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 647 has memoization hash e7d2dcd29cdd305c4d9aaa60adec061e parsl.dataflow.memoization INFO: Task 647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c5131a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 647 try 0 launched on executor multi with executor id 909 parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/3ca52e6a-1bab-47dd-9992-f13c688a233f_calibrate_898799_3.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/3ca52e6a-1bab-47dd-9992-f13c688a233f_calibrate_898799_3.stderr parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/3d2ef5e1-90f4-4971-94e2-b3127268545a_characterizeImage_898799_3.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/3d2ef5e1-90f4-4971-94e2-b3127268545a_characterizeImage_898799_3.stderr parsl.dataflow.dflow INFO: Task 559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b3e4de70a64a1e6ba008c1b1724b074 with result from task 559 parsl.dataflow.dflow DEBUG: Task 559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 560 has memoization hash 876fc71505db461da977284d74bd5a85 parsl.dataflow.memoization INFO: Task 560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 560 try 0 launched on executor multi with executor id 910 parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/4c0458a3-9709-4d33-958b-f1f92ad60840_calibrate_898695_59.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/4c0458a3-9709-4d33-958b-f1f92ad60840_calibrate_898695_59.stderr parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/4d15a0c9-4d09-4613-ab8b-6b95fcfca653_characterizeImage_898695_59.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/4d15a0c9-4d09-4613-ab8b-6b95fcfca653_characterizeImage_898695_59.stderr parsl.dataflow.dflow INFO: Task 421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46484aa14e27ea42b4c2a76484b74e20 with result from task 421 parsl.dataflow.dflow DEBUG: Task 421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 422 has memoization hash 321c769a36239c9953dc8ddf37abdcfd parsl.dataflow.memoization INFO: Task 422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4805e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 911 parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/18d9b4dd-c723-407d-9999-41d8481581dd_calibrate_898798_9.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/18d9b4dd-c723-407d-9999-41d8481581dd_calibrate_898798_9.stderr parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/80292448-a1f3-4c0b-aa1c-500df8ac5d4a_characterizeImage_898798_9.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/80292448-a1f3-4c0b-aa1c-500df8ac5d4a_characterizeImage_898798_9.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ce31b63de8a1e8db5b02d67bed0cd4b with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash fd88c9360c509fa50b01674b3d4029d9 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 912 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/185f8949-b778-4aea-b0b1-da124503f21d_calibrate_898798_53.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/185f8949-b778-4aea-b0b1-da124503f21d_calibrate_898798_53.stderr parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/840b8b6c-e6a1-4ba1-94f0-7b93c9bdd100_characterizeImage_898798_53.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/840b8b6c-e6a1-4ba1-94f0-7b93c9bdd100_characterizeImage_898798_53.stderr parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58828e96268da24db82213bc3794815a with result from task 220 parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:32 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash f942578253041c274046cd4b61af18da parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ef380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 913 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/6d8b84ad-d86a-4417-870e-1041bba36268_calibrate_898799_4.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/6d8b84ad-d86a-4417-870e-1041bba36268_calibrate_898799_4.stderr parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/88df0517-1992-4243-aebe-74b9cc2d561a_characterizeImage_898799_4.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/88df0517-1992-4243-aebe-74b9cc2d561a_characterizeImage_898799_4.stderr parsl.dataflow.dflow INFO: Task 880 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5967196e86dbcd02374105454419dbdc with result from task 880 parsl.dataflow.dflow DEBUG: Task 880 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 881 has memoization hash 90e3028ef7699cb4288b35bf55a097bb parsl.dataflow.memoization INFO: Task 881 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c442980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 881 try 0 launched on executor multi with executor id 914 parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/44226f37-12be-4bf5-b6c4-1fa1273ebff5_calibrate_898798_3.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/44226f37-12be-4bf5-b6c4-1fa1273ebff5_calibrate_898798_3.stderr parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ee8cb9d6-20ff-4625-a41b-e85548f00bbf_characterizeImage_898798_3.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ee8cb9d6-20ff-4625-a41b-e85548f00bbf_characterizeImage_898798_3.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 391 try 1 failed parsl.dataflow.dflow ERROR: Task 391 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 803724b20eb447b5fee3f20f596264d3 with result from task 391 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 392 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 392 try 0 failed parsl.dataflow.dflow INFO: Task 392 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/6a223240-35c7-4b5a-a44b-4cac925d5f0a_calibrate_898695_2.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/6a223240-35c7-4b5a-a44b-4cac925d5f0a_calibrate_898695_2.stderr parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/3143bd25-3fe9-4373-ad0c-e4d787503f78_characterizeImage_898695_2.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/3143bd25-3fe9-4373-ad0c-e4d787503f78_characterizeImage_898695_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 664 try 1 failed parsl.dataflow.dflow ERROR: Task 664 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry b39895c145d9c2f38f6a6458ac769dbb with result from task 664 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 665 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 665 try 0 failed parsl.dataflow.dflow INFO: Task 665 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/2b945c71-01f7-4249-b27d-b111d6d651ae_calibrate_898694_2.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/2b945c71-01f7-4249-b27d-b111d6d651ae_calibrate_898694_2.stderr parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/0d8e4103-985c-4101-9969-cc0402388124_characterizeImage_898694_2.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/0d8e4103-985c-4101-9969-cc0402388124_characterizeImage_898694_2.stderr parsl.dataflow.dflow INFO: Task 442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c826b0f2e6ba964cbd7ef1a3f77a2a9 with result from task 442 parsl.dataflow.dflow DEBUG: Task 442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 443 has memoization hash 825e57f9d13260095333cc6d58553367 parsl.dataflow.memoization INFO: Task 443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c481da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 443 try 0 launched on executor multi with executor id 915 parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/d98b3909-7102-45b6-845a-c340ad39ff03_calibrate_898798_50.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/d98b3909-7102-45b6-845a-c340ad39ff03_calibrate_898798_50.stderr parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/98045237-eff2-4650-97d8-0f37ca8a5a52_characterizeImage_898798_50.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/98045237-eff2-4650-97d8-0f37ca8a5a52_characterizeImage_898798_50.stderr parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1491530434f7c97ad515d90a8c350782 with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 778826edf612e36b143b81837c8995bd parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 916 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/18220d26-02dc-46a5-92ac-efb380e63ef8_calibrate_898799_18.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/18220d26-02dc-46a5-92ac-efb380e63ef8_calibrate_898799_18.stderr parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/9379e340-8cfa-41b7-a84c-9d4eeaa119d6_characterizeImage_898799_18.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/9379e340-8cfa-41b7-a84c-9d4eeaa119d6_characterizeImage_898799_18.stderr parsl.dataflow.dflow INFO: Task 844 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 887f7d8f1270253f3b462ca28ddb911e with result from task 844 parsl.dataflow.dflow DEBUG: Task 844 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 845 has memoization hash 792a7980eecd1d5aa4b66c5e5bd611b4 parsl.dataflow.memoization INFO: Task 845 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c440400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 845 try 0 launched on executor multi with executor id 917 parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/336c7a68-0841-4269-a5b4-6348483c723b_calibrate_898798_42.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/336c7a68-0841-4269-a5b4-6348483c723b_calibrate_898798_42.stderr parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/aca03799-8d72-4d70-b878-d6ee08834078_characterizeImage_898798_42.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/aca03799-8d72-4d70-b878-d6ee08834078_characterizeImage_898798_42.stderr parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9295d614d0ade23590ffe8702a1715a9 with result from task 397 parsl.dataflow.dflow DEBUG: Task 397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 398 has memoization hash f408570ac0543da2f52e963e649747f4 parsl.dataflow.memoization INFO: Task 398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d662ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 918 parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/0ebae809-8f3e-455d-b7d4-c4bcdc9d78d1_calibrate_898694_7.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/0ebae809-8f3e-455d-b7d4-c4bcdc9d78d1_calibrate_898694_7.stderr parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/3037ca79-76da-4e6c-9011-dde6c3de88ae_characterizeImage_898694_7.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/3037ca79-76da-4e6c-9011-dde6c3de88ae_characterizeImage_898694_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:34 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f6a91a4c853fdd3f0c8d0a3da2c1350 with result from task 598 parsl.dataflow.dflow DEBUG: Task 598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 599 has memoization hash 7828676b8f629a9f76954d6cebb90536 parsl.dataflow.memoization INFO: Task 599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dfe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 599 try 0 launched on executor multi with executor id 919 parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/8426e801-03e9-4255-8faa-7ca8d11f43ca_calibrate_898799_41.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/8426e801-03e9-4255-8faa-7ca8d11f43ca_calibrate_898799_41.stderr parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/33b8bc78-2410-4bae-8ef8-2c1afaf6aa8a_characterizeImage_898799_41.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/33b8bc78-2410-4bae-8ef8-2c1afaf6aa8a_characterizeImage_898799_41.stderr parsl.dataflow.dflow INFO: Task 556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97d69c38a15e2abdfc07830e034e09a8 with result from task 556 parsl.dataflow.dflow DEBUG: Task 556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 557 has memoization hash 5162e534511119e18d9eb52cf0895183 parsl.dataflow.memoization INFO: Task 557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dd4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 557 try 0 launched on executor multi with executor id 920 parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/58e14350-bc40-4e2f-87a3-550c6939bfae_calibrate_898800_42.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/58e14350-bc40-4e2f-87a3-550c6939bfae_calibrate_898800_42.stderr parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/4fbd67e7-8618-4a5b-b265-0d84ff3cf96f_characterizeImage_898800_42.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/4fbd67e7-8618-4a5b-b265-0d84ff3cf96f_characterizeImage_898800_42.stderr parsl.dataflow.dflow INFO: Task 847 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be386370dda759a42ce0fed9ee5f75a4 with result from task 847 parsl.dataflow.dflow DEBUG: Task 847 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 848 has memoization hash 626b766b86fbba0a1047c5511921cac3 parsl.dataflow.memoization INFO: Task 848 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c440720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 848 try 0 launched on executor multi with executor id 921 parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/4b0a708e-75c4-4977-bff3-7185722124c3_calibrate_898695_52.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/4b0a708e-75c4-4977-bff3-7185722124c3_calibrate_898695_52.stderr parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/877730e7-9035-4461-a080-15fbd738c8c1_characterizeImage_898695_52.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/877730e7-9035-4461-a080-15fbd738c8c1_characterizeImage_898695_52.stderr parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3203c0344fa1da7673d9866178eff350 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/6cff600c-4bfd-4346-9f8d-48b61977e1f0_calibrate_898694_53.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/6cff600c-4bfd-4346-9f8d-48b61977e1f0_calibrate_898694_53.stderr parsl.dataflow.dflow INFO: Task 485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3effc77ce92a4ea844bb68ead9e15fd with result from task 485 parsl.dataflow.dflow DEBUG: Task 485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/29d2352d-173b-4550-af9d-10db86c150e2_calibrate_898696_31.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/29d2352d-173b-4550-af9d-10db86c150e2_calibrate_898696_31.stderr parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf4ef0c6d8dcf2aca95e172c2177720d with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash af67fbf8eb9abc6a5df2c4a72cb08785 parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cf1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 922 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/a9b9b203-03db-47de-a126-74b8a1bc2434_calibrate_898695_4.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/a9b9b203-03db-47de-a126-74b8a1bc2434_calibrate_898695_4.stderr parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/f980714d-2919-4239-bee9-e7503e49a96a_characterizeImage_898695_4.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/f980714d-2919-4239-bee9-e7503e49a96a_characterizeImage_898695_4.stderr parsl.dataflow.dflow INFO: Task 505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c4844f4052f89f292c94da53fee16bf with result from task 505 parsl.dataflow.dflow DEBUG: Task 505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 506 has memoization hash 68f488310c05882531bfd00b896c5f15 parsl.dataflow.memoization INFO: Task 506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 923 parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/33486b45-81b7-4c99-876a-1ce51f88cd31_calibrate_898800_37.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/33486b45-81b7-4c99-876a-1ce51f88cd31_calibrate_898800_37.stderr parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/a6152900-2763-4d37-bbed-803a41b064f2_characterizeImage_898800_37.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/a6152900-2763-4d37-bbed-803a41b064f2_characterizeImage_898800_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:35 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 564e2a67468e0bb462a2980c9144010f with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/934390cc-cb46-435a-bfb6-c210af01e674_calibrate_898694_57.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/934390cc-cb46-435a-bfb6-c210af01e674_calibrate_898694_57.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ffd425ff914452438e66c03e0f58a1b with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/fd362619-6014-4122-8a6f-62f5916bb2f4_calibrate_898696_42.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/fd362619-6014-4122-8a6f-62f5916bb2f4_calibrate_898696_42.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3c0b1a4e00a2ac3df3b1bcad1560679 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/735355a5-f5c5-49f1-907c-798f99d54351_calibrate_898695_62.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/735355a5-f5c5-49f1-907c-798f99d54351_calibrate_898695_62.stderr parsl.dataflow.dflow INFO: Task 862 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2faa22978d559de173ea698c590413ba with result from task 862 parsl.dataflow.dflow DEBUG: Task 862 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 863 has memoization hash 93871cbf470e484e1a5a0df7c175a2df parsl.dataflow.memoization INFO: Task 863 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4416c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 863 try 0 launched on executor multi with executor id 924 parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/8e55252a-d2b3-499b-b816-7da745f9001b_calibrate_898696_7.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/8e55252a-d2b3-499b-b816-7da745f9001b_calibrate_898696_7.stderr parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/d9fc4a9c-cdcf-46c3-96d9-bfcdff34eb4f_characterizeImage_898696_7.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/d9fc4a9c-cdcf-46c3-96d9-bfcdff34eb4f_characterizeImage_898696_7.stderr parsl.dataflow.dflow INFO: Task 541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89c047a518e8286f41524ccc30b53807 with result from task 541 parsl.dataflow.dflow DEBUG: Task 541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 542 has memoization hash f4cfb237232501a78b32cda3ef38321d parsl.dataflow.memoization INFO: Task 542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 542 try 0 launched on executor multi with executor id 925 parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/ec1f83d5-a716-4bde-b330-141853d2cf87_calibrate_898800_20.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/ec1f83d5-a716-4bde-b330-141853d2cf87_calibrate_898800_20.stderr parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/24fccec6-1b9f-4118-ac9b-436178150dd5_characterizeImage_898800_20.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/24fccec6-1b9f-4118-ac9b-436178150dd5_characterizeImage_898800_20.stderr parsl.dataflow.dflow INFO: Task 1057 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dedd27e2c7e264f75e50624d4bc18e01 with result from task 1057 parsl.dataflow.dflow DEBUG: Task 1057 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1058 has memoization hash e74c88e975c4db8e4402fa2c32d0657e parsl.dataflow.memoization INFO: Task 1058 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c328720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1058 try 0 launched on executor multi with executor id 926 parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/9122fb1e-5828-46cd-b4f4-b9acab8e0d18_calibrate_898798_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/9122fb1e-5828-46cd-b4f4-b9acab8e0d18_calibrate_898798_48.stderr parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/c1107c1d-285a-440f-8802-b6685b40a352_characterizeImage_898798_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/c1107c1d-285a-440f-8802-b6685b40a352_characterizeImage_898798_48.stderr parsl.dataflow.dflow INFO: Task 1114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18b621225e636b3978c6e5b95a799da9 with result from task 1114 parsl.dataflow.dflow DEBUG: Task 1114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1115 has memoization hash 3b024ce4023e23df1cdcb653bc610a5a parsl.dataflow.memoization INFO: Task 1115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c351e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1115 try 0 launched on executor multi with executor id 927 parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/440fc355-368e-46c1-9115-f7eb8f958d50_calibrate_898799_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/440fc355-368e-46c1-9115-f7eb8f958d50_calibrate_898799_20.stderr parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/2ede117f-f90a-4e69-b281-7d238015d6a1_characterizeImage_898799_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/2ede117f-f90a-4e69-b281-7d238015d6a1_characterizeImage_898799_20.stderr parsl.dataflow.dflow INFO: Task 1054 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99383185b307567987c2f88af3c6ec3b with result from task 1054 parsl.dataflow.dflow DEBUG: Task 1054 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1055 has memoization hash 087d210ab0934db19a6f60fa57ada051 parsl.dataflow.memoization INFO: Task 1055 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c350040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1055 try 0 launched on executor multi with executor id 928 parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/61d049ea-ee5f-4342-bbe9-884ca8b5b4cd_calibrate_898800_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/61d049ea-ee5f-4342-bbe9-884ca8b5b4cd_calibrate_898800_55.stderr parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/d482ea6f-1bb7-4b7d-8534-bc1fe5ea4cd1_characterizeImage_898800_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/d482ea6f-1bb7-4b7d-8534-bc1fe5ea4cd1_characterizeImage_898800_55.stderr parsl.dataflow.dflow INFO: Task 994 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b60ed8b9a2905e3b06528c740b8e67ae with result from task 994 parsl.dataflow.dflow DEBUG: Task 994 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 995 has memoization hash a0007b0a4bd4c3234e043bec1a59bbd8 parsl.dataflow.memoization INFO: Task 995 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2da0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 995 try 0 launched on executor multi with executor id 929 parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/fe723d74-e386-4b53-99d5-35d50ac2cc83_calibrate_898798_7.stdout parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/fe723d74-e386-4b53-99d5-35d50ac2cc83_calibrate_898798_7.stderr parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ba336da1-e362-4571-a751-659e82b0d625_characterizeImage_898798_7.stdout parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ba336da1-e362-4571-a751-659e82b0d625_characterizeImage_898798_7.stderr parsl.dataflow.dflow INFO: Task 619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c861ce7e8809b9d786381b7531ea838d with result from task 619 parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:36 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow DEBUG: Task 619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 620 has memoization hash fc6d6c044adfe25e5f76f960bd705932 parsl.dataflow.memoization INFO: Task 620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c5118a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 620 try 0 launched on executor multi with executor id 930 parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/2e42dd95-8021-463d-98f9-761ef163ba9f_calibrate_898800_38.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/2e42dd95-8021-463d-98f9-761ef163ba9f_calibrate_898800_38.stderr parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/e1dfe223-522c-410f-895f-4e44e59b736a_characterizeImage_898800_38.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/e1dfe223-522c-410f-895f-4e44e59b736a_characterizeImage_898800_38.stderr parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f13230a5928c6f332491c38437a33b53 with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash 595cca24b6eaa46a295b2785d193a262 parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 931 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/856ea1ad-e8d9-4f15-8975-64aeab8e2952_calibrate_898696_59.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/856ea1ad-e8d9-4f15-8975-64aeab8e2952_calibrate_898696_59.stderr parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/9196f4db-d2e5-4cb1-81ec-8a08791adda2_characterizeImage_898696_59.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/9196f4db-d2e5-4cb1-81ec-8a08791adda2_characterizeImage_898696_59.stderr parsl.dataflow.dflow INFO: Task 571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb26be1d1d7ff6290f152c7996aeceeb with result from task 571 parsl.dataflow.dflow DEBUG: Task 571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 572 has memoization hash 86cad469e35658f3cc859021e08ed78f parsl.dataflow.memoization INFO: Task 572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4de160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 572 try 0 launched on executor multi with executor id 932 parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/11c2656e-7c3a-4b74-bb0f-cb52ac438f1c_calibrate_898799_26.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/11c2656e-7c3a-4b74-bb0f-cb52ac438f1c_calibrate_898799_26.stderr parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/ba463b6a-e421-49d4-96f4-b4fec073c5ac_characterizeImage_898799_26.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/ba463b6a-e421-49d4-96f4-b4fec073c5ac_characterizeImage_898799_26.stderr parsl.dataflow.dflow INFO: Task 628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c58dd320d85787eb54949d1eba2bc316 with result from task 628 parsl.dataflow.dflow DEBUG: Task 628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 629 has memoization hash 3ca74f45a2e1bac4bce88c0083ac3395 parsl.dataflow.memoization INFO: Task 629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c512200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 629 try 0 launched on executor multi with executor id 933 parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/4d5a717f-48fa-4bef-adfd-ca7094cef6b4_calibrate_898799_34.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/4d5a717f-48fa-4bef-adfd-ca7094cef6b4_calibrate_898799_34.stderr parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/3785309e-d272-4d88-bb97-415daec530d3_characterizeImage_898799_34.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/3785309e-d272-4d88-bb97-415daec530d3_characterizeImage_898799_34.stderr parsl.dataflow.dflow INFO: Task 649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9184ce6e5ac2753d9ec0ad5e6c4f1217 with result from task 649 parsl.dataflow.dflow DEBUG: Task 649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 650 has memoization hash f9eae98ca9224320399426d9d708d5d4 parsl.dataflow.memoization INFO: Task 650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c5136a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 650 try 0 launched on executor multi with executor id 934 parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/102c168a-1de2-467f-a08f-30c9b105f6f7_calibrate_898798_18.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/102c168a-1de2-467f-a08f-30c9b105f6f7_calibrate_898798_18.stderr parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/5b7518bf-ecca-445d-8280-79b6393113c1_characterizeImage_898798_18.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/5b7518bf-ecca-445d-8280-79b6393113c1_characterizeImage_898798_18.stderr parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b63cc58e176875bf90e977f54383af4 with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/d1b50ea5-55b9-44dc-99e7-1d53d0008530_calibrate_898694_42.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/d1b50ea5-55b9-44dc-99e7-1d53d0008530_calibrate_898694_42.stderr parsl.dataflow.dflow INFO: Task 961 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74e0e85d087cef490e722168c7f21432 with result from task 961 parsl.dataflow.dflow DEBUG: Task 961 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 962 has memoization hash 7fa76147d5f4e98df58f18cc9e068894 parsl.dataflow.memoization INFO: Task 962 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2932e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 962 try 0 launched on executor multi with executor id 935 parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/d4c3bb47-7d85-4981-910f-36e5917bdb47_calibrate_898800_3.stdout parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/d4c3bb47-7d85-4981-910f-36e5917bdb47_calibrate_898800_3.stderr parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/1cb94f31-9003-4445-a4d9-c924b45d3f5c_characterizeImage_898800_3.stdout parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/1cb94f31-9003-4445-a4d9-c924b45d3f5c_characterizeImage_898800_3.stderr parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 753817b9665d0036831e69cca439d5a9 with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/0061690f-b2dc-4c25-82ed-1df158ed7e06_calibrate_898800_31.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/0061690f-b2dc-4c25-82ed-1df158ed7e06_calibrate_898800_31.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8133e8839f59a425f20c475716677e51 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/f2812508-890d-423c-ae36-62e33959fe45_calibrate_898694_62.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/f2812508-890d-423c-ae36-62e33959fe45_calibrate_898694_62.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63a950ddb84a6421892977f52690af4e with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 1993466b4b89a0066597dd7f50acd4bc parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5aafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 936 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/6fb44d5e-588a-4d92-b63f-f351838f79e5_calibrate_898696_20.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/6fb44d5e-588a-4d92-b63f-f351838f79e5_calibrate_898696_20.stderr parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/ba606731-dd0a-46c7-abeb-9a6b74c6ad7a_characterizeImage_898696_20.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/ba606731-dd0a-46c7-abeb-9a6b74c6ad7a_characterizeImage_898696_20.stderr parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 725019a58421836aaa5b7f22d2cd44ae with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/b913b389-7f8a-4427-8f77-a275b23101cd_calibrate_898696_21.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/b913b389-7f8a-4427-8f77-a275b23101cd_calibrate_898696_21.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5515f18c2e3d31d45e2e92243f7915f8 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/01365429-6f46-4101-833a-4c7ca79eaf7f_calibrate_898695_47.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/01365429-6f46-4101-833a-4c7ca79eaf7f_calibrate_898695_47.stderr parsl.dataflow.dflow INFO: Task 1051 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2fd536113ba461af81c3a7f5e9d6feb with result from task 1051 parsl.dataflow.dflow DEBUG: Task 1051 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1052 has memoization hash 1e41336a2472a7f8a0cc5140c3a75604 parsl.dataflow.memoization INFO: Task 1052 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1052 try 0 launched on executor multi with executor id 937 parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/e42dbf8d-bbe0-4398-af78-96ee9fb90d83_calibrate_898800_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/e42dbf8d-bbe0-4398-af78-96ee9fb90d83_calibrate_898800_9.stderr parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/758c13da-3894-4988-9dda-f4358093d5d4_characterizeImage_898800_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/758c13da-3894-4988-9dda-f4358093d5d4_characterizeImage_898800_9.stderr parsl.dataflow.dflow INFO: Task 1063 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa1c2006897f3a52b2eb6024bfc1afb9 with result from task 1063 parsl.dataflow.dflow DEBUG: Task 1063 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1064 has memoization hash c525115006ad3f2af78990eb5e7c5cc7 parsl.dataflow.memoization INFO: Task 1064 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c328ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1064 try 0 launched on executor multi with executor id 938 parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/b9785b86-4018-4c12-ba5c-4d02f8deca3b_calibrate_898695_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/b9785b86-4018-4c12-ba5c-4d02f8deca3b_calibrate_898695_9.stderr parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/dd26452c-0598-445c-b2af-262387565937_characterizeImage_898695_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/dd26452c-0598-445c-b2af-262387565937_characterizeImage_898695_9.stderr parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd8c1c5510a32e1a6b7b055d199c80d8 with result from task 401 parsl.dataflow.dflow DEBUG: Task 401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/5988dc4c-4381-421a-9a10-7d9ceb990106_calibrate_898694_22.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/5988dc4c-4381-421a-9a10-7d9ceb990106_calibrate_898694_22.stderr parsl.dataflow.dflow INFO: Task 526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 993fccca5b86d3f1c0803e8366ba11d3 with result from task 526 parsl.dataflow.dflow DEBUG: Task 526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 527 has memoization hash 87d3eb04140b2039543732483679f5bb parsl.dataflow.memoization INFO: Task 527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 527 try 0 launched on executor multi with executor id 939 parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/b05e4c1f-55ea-41b5-9752-a0e2d97e5718_calibrate_898694_17.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/b05e4c1f-55ea-41b5-9752-a0e2d97e5718_calibrate_898694_17.stderr parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/59196439-7766-4447-8f1c-be4e3e692698_characterizeImage_898694_17.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/59196439-7766-4447-8f1c-be4e3e692698_characterizeImage_898694_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:37 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 823f9fac37660abe9e908ec46c47bdec with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/8a1ab9fa-3f1b-4a5d-ade1-1f5024240b20_calibrate_898694_18.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/8a1ab9fa-3f1b-4a5d-ade1-1f5024240b20_calibrate_898694_18.stderr parsl.dataflow.dflow INFO: Task 470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cea5a5b59c4f9869387fe0a6e24d37d with result from task 470 parsl.dataflow.dflow DEBUG: Task 470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/1bf5a92a-8c33-43bc-960a-ed52abb5655a_calibrate_898798_31.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/1bf5a92a-8c33-43bc-960a-ed52abb5655a_calibrate_898798_31.stderr parsl.dataflow.dflow INFO: Task 1099 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b41640edb91a52d9a2b04d6a49c9568 with result from task 1099 parsl.dataflow.dflow DEBUG: Task 1099 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1100 has memoization hash 5315802b5ac4163b8af7327d07716eaa parsl.dataflow.memoization INFO: Task 1100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c350ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1100 try 0 launched on executor multi with executor id 940 parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/8be85f31-09d4-4e82-b21d-027897835edb_calibrate_898800_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/8be85f31-09d4-4e82-b21d-027897835edb_calibrate_898800_50.stderr parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/5be748ac-adbd-40fb-a77a-e6d235d47721_characterizeImage_898800_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/5be748ac-adbd-40fb-a77a-e6d235d47721_characterizeImage_898800_50.stderr parsl.dataflow.dflow INFO: Task 1015 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98c247df475aa743c7d92ed9eaf5f5d1 with result from task 1015 parsl.dataflow.dflow DEBUG: Task 1015 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1016 has memoization hash 8e5132214c8bb6b55e5903f4dd5f53bf parsl.dataflow.memoization INFO: Task 1016 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2db4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1016 try 0 launched on executor multi with executor id 941 parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/efefc570-29c2-424a-82af-c495dfc81f20_calibrate_898696_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/efefc570-29c2-424a-82af-c495dfc81f20_calibrate_898696_45.stderr parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/ec930468-25a3-40b8-99e4-143b6ee33931_characterizeImage_898696_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/ec930468-25a3-40b8-99e4-143b6ee33931_characterizeImage_898696_45.stderr parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5dd10254b69b3f7caaffa174e0d6bd0d with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 4a89b993a9de4fbc2bc06587f5519482 parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ee3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 942 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/e3aed31e-40bd-4c61-9a5d-64856ecbf96a_calibrate_898695_41.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/e3aed31e-40bd-4c61-9a5d-64856ecbf96a_calibrate_898695_41.stderr parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/0ba9eeb6-f72e-4fb8-8221-87b399700808_characterizeImage_898695_41.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/0ba9eeb6-f72e-4fb8-8221-87b399700808_characterizeImage_898695_41.stderr parsl.dataflow.dflow INFO: Task 643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8730a5b43d3b915447fb42ebcd60e25 with result from task 643 parsl.dataflow.dflow DEBUG: Task 643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 644 has memoization hash e2edda1b4886e1e77af95bc8532d963b parsl.dataflow.memoization INFO: Task 644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c512fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 644 try 0 launched on executor multi with executor id 943 parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/4faadd51-c882-4f86-9de9-fcf3aaa0ed8a_calibrate_898798_11.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/4faadd51-c882-4f86-9de9-fcf3aaa0ed8a_calibrate_898798_11.stderr parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/8e9ea480-804b-4c2d-a1f0-1ca1c6a37114_characterizeImage_898798_11.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/8e9ea480-804b-4c2d-a1f0-1ca1c6a37114_characterizeImage_898798_11.stderr parsl.dataflow.dflow INFO: Task 832 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f01dc5a8511875c24f1455847bcc5d8c with result from task 832 parsl.dataflow.dflow DEBUG: Task 832 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 833 has memoization hash 27f92430df6abd6bdc511dd841001be8 parsl.dataflow.memoization INFO: Task 833 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 833 try 0 launched on executor multi with executor id 944 parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/15537074-b6d6-4793-a6aa-6de8bf8addb4_calibrate_898694_26.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/15537074-b6d6-4793-a6aa-6de8bf8addb4_calibrate_898694_26.stderr parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/8c899deb-43d3-49af-957a-499befb13ffc_characterizeImage_898694_26.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/8c899deb-43d3-49af-957a-499befb13ffc_characterizeImage_898694_26.stderr parsl.dataflow.dflow INFO: Task 721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec0635f2e2f8fa384a918d34f6bb5a8f with result from task 721 parsl.dataflow.dflow DEBUG: Task 721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 722 has memoization hash e49a08fb477b3eafc42d7fabb302b499 parsl.dataflow.memoization INFO: Task 722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 722 try 0 launched on executor multi with executor id 945 parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/b8f5c514-fc37-4654-b10f-fa555465899c_calibrate_898696_6.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/b8f5c514-fc37-4654-b10f-fa555465899c_calibrate_898696_6.stderr parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/705781c6-5678-404f-b920-8dc5bc0d2ab4_characterizeImage_898696_6.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/705781c6-5678-404f-b920-8dc5bc0d2ab4_characterizeImage_898696_6.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b78463ea5f552d541641ea1f5f391932 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/9ca2c801-ddfe-419c-94b8-1dfdcc7e4a9b_calibrate_898696_36.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/9ca2c801-ddfe-419c-94b8-1dfdcc7e4a9b_calibrate_898696_36.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 540edd5fbf28d4c0eb75c315e83e94df with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/51da929e-70e7-4d64-ac8e-8919967fa8fa_calibrate_898694_47.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/51da929e-70e7-4d64-ac8e-8919967fa8fa_calibrate_898694_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:39 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23f85b21ec57e21da129e8b3390d66e8 with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/edb4db20-eb85-4ddc-ac72-0e6ec9f4ae51_calibrate_898694_10.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/edb4db20-eb85-4ddc-ac72-0e6ec9f4ae51_calibrate_898694_10.stderr parsl.dataflow.dflow INFO: Task 997 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 757b2310d9f2984b358e3f2423b79800 with result from task 997 parsl.dataflow.dflow DEBUG: Task 997 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 998 has memoization hash c2f3dfae81c69787cec07fb135436569 parsl.dataflow.memoization INFO: Task 998 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2da3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 998 try 0 launched on executor multi with executor id 946 parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/cc9c34de-095b-4f01-a002-f5db16236fe2_calibrate_898798_19.stdout parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/cc9c34de-095b-4f01-a002-f5db16236fe2_calibrate_898798_19.stderr parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/89ea6120-78ff-4c59-819b-33a55cc30e05_characterizeImage_898798_19.stdout parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/89ea6120-78ff-4c59-819b-33a55cc30e05_characterizeImage_898798_19.stderr parsl.dataflow.dflow INFO: Task 697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0eb7d0c690bbb5b89c5606d34fdd2e5c with result from task 697 parsl.dataflow.dflow DEBUG: Task 697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 698 has memoization hash 3e465d334b4a6db9696a64313c531a51 parsl.dataflow.memoization INFO: Task 698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3762a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 698 try 0 launched on executor multi with executor id 947 parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/4a2f70a0-ae2e-4105-8332-4e4906775c57_calibrate_898800_52.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/4a2f70a0-ae2e-4105-8332-4e4906775c57_calibrate_898800_52.stderr parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/17efda62-0adf-4afe-8e6b-d2a6772f176f_characterizeImage_898800_52.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/17efda62-0adf-4afe-8e6b-d2a6772f176f_characterizeImage_898800_52.stderr parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b972da7e648f8dcf849e4934297a7e6 with result from task 346 parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash 08168256ac451cc5efdb5f4486694f51 parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 948 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/2a08d6f7-5920-4ef8-b35c-8485fbbcf827_calibrate_898799_27.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/2a08d6f7-5920-4ef8-b35c-8485fbbcf827_calibrate_898799_27.stderr parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/abf42ef6-80ee-4ac7-873b-b2f6e1c362c9_characterizeImage_898799_27.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/abf42ef6-80ee-4ac7-873b-b2f6e1c362c9_characterizeImage_898799_27.stderr parsl.dataflow.dflow INFO: Task 1042 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70baa38865c63c6a162c33043362febc with result from task 1042 parsl.dataflow.dflow DEBUG: Task 1042 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1043 has memoization hash 4a55a5da6afbf1784baece4e22fe8572 parsl.dataflow.memoization INFO: Task 1043 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1043 try 0 launched on executor multi with executor id 949 parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/60947f7a-dae1-415d-a8bd-5653d541c143_calibrate_898798_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/60947f7a-dae1-415d-a8bd-5653d541c143_calibrate_898798_27.stderr parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/5a750542-aad2-4c5b-9654-8ce69a9633e4_characterizeImage_898798_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/5a750542-aad2-4c5b-9654-8ce69a9633e4_characterizeImage_898798_27.stderr parsl.dataflow.dflow INFO: Task 802 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73a8541a0c28358c8cb824dd22b0b426 with result from task 802 parsl.dataflow.dflow DEBUG: Task 802 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 803 has memoization hash 7e6b8c4f406f4451b3a6c485285f919d parsl.dataflow.memoization INFO: Task 803 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 803 try 0 launched on executor multi with executor id 950 parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/1d419d03-2cdb-4d47-83a6-afdb4647f699_calibrate_898694_32.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/1d419d03-2cdb-4d47-83a6-afdb4647f699_calibrate_898694_32.stderr parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/b3efbc14-5c18-4e2b-9491-e0294b592d00_characterizeImage_898694_32.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/b3efbc14-5c18-4e2b-9491-e0294b592d00_characterizeImage_898694_32.stderr parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a649392de091357d694f22746804dfe with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash 36c030459edcdb4fc8788be3c87790c5 parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 951 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/153f61da-e733-4240-8850-7382fa769764_calibrate_898695_26.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/153f61da-e733-4240-8850-7382fa769764_calibrate_898695_26.stderr parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/e837ed91-5bf2-41d7-be2f-18e8220b17be_characterizeImage_898695_26.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/e837ed91-5bf2-41d7-be2f-18e8220b17be_characterizeImage_898695_26.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a9b478f4c0dc98203e8d75975fd8e1b with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash a3c7d70730b0b9763e4d3cee3c5d528d parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d585800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 952 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/3785c97b-0b7b-4fc5-81e3-83822485c2bc_calibrate_898694_39.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/3785c97b-0b7b-4fc5-81e3-83822485c2bc_calibrate_898694_39.stderr parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/fe975d71-25e4-4fd3-89ff-a290c3e63a11_characterizeImage_898694_39.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/fe975d71-25e4-4fd3-89ff-a290c3e63a11_characterizeImage_898694_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:40 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f054771da2eb5812e7803163d80b628 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/4f959a19-e0fb-4175-ba41-544090c91a89_calibrate_898695_35.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/4f959a19-e0fb-4175-ba41-544090c91a89_calibrate_898695_35.stderr parsl.dataflow.dflow INFO: Task 475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7095f1c19f0176a35bf899f268c6b121 with result from task 475 parsl.dataflow.dflow DEBUG: Task 475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 476 has memoization hash 4fc50da008ca41f1647693b9e34874e7 parsl.dataflow.memoization INFO: Task 476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4828e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 476 try 0 launched on executor multi with executor id 953 parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/19bf2d7c-b097-4c8b-8b3d-56ff61deb672_calibrate_898800_48.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/19bf2d7c-b097-4c8b-8b3d-56ff61deb672_calibrate_898800_48.stderr parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/b7fb183b-a3ef-49a7-a0d2-ade2e10eb4ca_characterizeImage_898800_48.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/b7fb183b-a3ef-49a7-a0d2-ade2e10eb4ca_characterizeImage_898800_48.stderr parsl.dataflow.dflow INFO: Task 1024 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5de56370d8b78a0b569ce347e5a686d9 with result from task 1024 parsl.dataflow.dflow DEBUG: Task 1024 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1025 has memoization hash 220659b91ddc66d005a526aa72ee62dd parsl.dataflow.memoization INFO: Task 1025 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2dbf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1025 try 0 launched on executor multi with executor id 954 parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/4f3aa110-e615-47f8-8970-77fa3a8f4e6d_calibrate_898800_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/4f3aa110-e615-47f8-8970-77fa3a8f4e6d_calibrate_898800_6.stderr parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/92676c99-cc35-4d10-9bcb-29db7391e5a8_characterizeImage_898800_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/92676c99-cc35-4d10-9bcb-29db7391e5a8_characterizeImage_898800_6.stderr parsl.dataflow.dflow INFO: Task 683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c4bc9520bf83b6fbb2921d06d72fa8d with result from task 683 parsl.dataflow.dflow DEBUG: Task 683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/754d2c16-e782-4c12-abeb-8ec38cab3f1c_calibrate_898799_31.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/754d2c16-e782-4c12-abeb-8ec38cab3f1c_calibrate_898799_31.stderr parsl.dataflow.dflow INFO: Task 710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ceee281bca13545cc08e24636953e0ed with result from task 710 parsl.dataflow.dflow DEBUG: Task 710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/b812b617-2bce-4db0-b4cf-ca4cb1beeb5d_calibrate_898696_57.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/b812b617-2bce-4db0-b4cf-ca4cb1beeb5d_calibrate_898696_57.stderr parsl.dataflow.dflow INFO: Task 817 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c3286b177f0a595720fe2f4d02f742a with result from task 817 parsl.dataflow.dflow DEBUG: Task 817 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 818 has memoization hash 0a3de713a6b08f729e361760de5f5a1e parsl.dataflow.memoization INFO: Task 818 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 818 try 0 launched on executor multi with executor id 955 parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/1f0b242d-03d9-4e1f-846c-677e7cbbca31_calibrate_898799_35.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/1f0b242d-03d9-4e1f-846c-677e7cbbca31_calibrate_898799_35.stderr parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/153c99c6-13b1-4ca0-9ecc-adecae112988_characterizeImage_898799_35.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/153c99c6-13b1-4ca0-9ecc-adecae112988_characterizeImage_898799_35.stderr parsl.dataflow.dflow INFO: Task 703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfdad89e8063d10156f94bfb7c872678 with result from task 703 parsl.dataflow.dflow DEBUG: Task 703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 704 has memoization hash 87ce344c6b4bd9be19c2f8c1fdd3beb2 parsl.dataflow.memoization INFO: Task 704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c375bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 704 try 0 launched on executor multi with executor id 956 parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/3e7c19dd-3837-4d47-a021-751233c28358_calibrate_898798_35.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/3e7c19dd-3837-4d47-a021-751233c28358_calibrate_898798_35.stderr parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/364c5ceb-bb35-48aa-94ba-026f29b334b9_characterizeImage_898798_35.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/364c5ceb-bb35-48aa-94ba-026f29b334b9_characterizeImage_898798_35.stderr parsl.dataflow.dflow INFO: Task 1111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf5774f2f7f0b2b39906e6f235bb76be with result from task 1111 parsl.dataflow.dflow DEBUG: Task 1111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1112 has memoization hash 5700c8fbec033a5df100d2d7f218254d parsl.dataflow.memoization INFO: Task 1112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c351b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1112 try 0 launched on executor multi with executor id 957 parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/7a13085d-1e1e-44dd-9449-59a0bc7fd0b0_calibrate_898696_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/7a13085d-1e1e-44dd-9449-59a0bc7fd0b0_calibrate_898696_56.stderr parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/91cb77ca-d098-4e3d-a37d-ded2a6fc4fd5_characterizeImage_898696_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/91cb77ca-d098-4e3d-a37d-ded2a6fc4fd5_characterizeImage_898696_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:41 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry debd4bccdafafa4fe6f893187e6d1a5e with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 412e41fc509bb567cef498b6972114cb parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ed120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 958 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/d2ad87e6-fd09-43dd-9ee9-2385317df624_calibrate_898696_5.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/d2ad87e6-fd09-43dd-9ee9-2385317df624_calibrate_898696_5.stderr parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/a5ce4e97-5c9f-49b9-8b69-b48bfe1e953f_characterizeImage_898696_5.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/a5ce4e97-5c9f-49b9-8b69-b48bfe1e953f_characterizeImage_898696_5.stderr parsl.dataflow.dflow INFO: Task 385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91c4defa0234fbc0346a8ef188b282a4 with result from task 385 parsl.dataflow.dflow DEBUG: Task 385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 386 has memoization hash 783c126fd67ea9f9317890f56a1d8097 parsl.dataflow.memoization INFO: Task 386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d662020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 959 parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/e66cc113-8142-46f7-ab2f-f3e4fea42a03_calibrate_898800_26.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/e66cc113-8142-46f7-ab2f-f3e4fea42a03_calibrate_898800_26.stderr parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/491098e3-1072-443d-b1f7-7553527ef2f4_characterizeImage_898800_26.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/491098e3-1072-443d-b1f7-7553527ef2f4_characterizeImage_898800_26.stderr parsl.dataflow.dflow INFO: Task 508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 565ce4012fd324c4d9a16e0f3513f25d with result from task 508 parsl.dataflow.dflow DEBUG: Task 508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 509 has memoization hash af458c1b888f8c4e3992266b509339a8 parsl.dataflow.memoization INFO: Task 509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 509 try 0 launched on executor multi with executor id 960 parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/75da0acb-6ca5-46bf-afa6-12c20917cc52_calibrate_898799_39.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/75da0acb-6ca5-46bf-afa6-12c20917cc52_calibrate_898799_39.stderr parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/1d2f74a7-7c94-4755-8a1a-8d084c00712b_characterizeImage_898799_39.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/1d2f74a7-7c94-4755-8a1a-8d084c00712b_characterizeImage_898799_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 346 parsl.dataflow.strategy DEBUG: Executor multi has 346 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 2024-03-13 09:48:42 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 09:48:42 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f415fa9b646e456a95bea7aed006a98 with result from task 424 parsl.dataflow.dflow DEBUG: Task 424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 425 has memoization hash daac6ad3e6e1762da6adbb28520bf2ca parsl.dataflow.memoization INFO: Task 425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c480900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 961 parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/776c9241-f0da-479c-b466-5a673a5faad6_calibrate_898798_30.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/776c9241-f0da-479c-b466-5a673a5faad6_calibrate_898798_30.stderr parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ce197a9b-543e-445b-9a12-63fe75b05d8e_characterizeImage_898798_30.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ce197a9b-543e-445b-9a12-63fe75b05d8e_characterizeImage_898798_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:43 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ffa337185f55e3f82ee8395f69bf291 with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/adb2ab80-579f-4c08-b9b5-608add0b59b2_calibrate_898696_35.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/adb2ab80-579f-4c08-b9b5-608add0b59b2_calibrate_898696_35.stderr parsl.dataflow.dflow INFO: Task 787 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66b1a60f456d0b4f89a25acafb32f92c with result from task 787 parsl.dataflow.dflow DEBUG: Task 787 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 788 has memoization hash af6f9c7b31924800887ed4321a9f201a parsl.dataflow.memoization INFO: Task 788 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3bb4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 788 try 0 launched on executor multi with executor id 962 parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/08d03992-9eab-4679-b25b-c89b9b8d07a4_calibrate_898799_16.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/08d03992-9eab-4679-b25b-c89b9b8d07a4_calibrate_898799_16.stderr parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/153275a5-848e-404f-85fb-fdb2d5f5e3d7_characterizeImage_898799_16.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/153275a5-848e-404f-85fb-fdb2d5f5e3d7_characterizeImage_898799_16.stderr parsl.dataflow.dflow INFO: Task 973 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63c58605f12a45ecd8b0600be0904240 with result from task 973 parsl.dataflow.dflow DEBUG: Task 973 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 974 has memoization hash e1deab933c93afc3da0a31590d4a0f6b parsl.dataflow.memoization INFO: Task 974 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 974 try 0 launched on executor multi with executor id 963 parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/40b03996-a623-46aa-92ab-960c92c0c6cb_calibrate_898696_15.stdout parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/40b03996-a623-46aa-92ab-960c92c0c6cb_calibrate_898696_15.stderr parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/71d942ea-2198-413c-90c8-62e4834076a7_characterizeImage_898696_15.stdout parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/71d942ea-2198-413c-90c8-62e4834076a7_characterizeImage_898696_15.stderr parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05e5d819800eb778c47782406b81ecfd with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 5e887dd1842e3a6bc8e7b50a982fd4a5 parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 964 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/1c0b1cbd-4917-44b4-b859-d2b9059cf389_calibrate_898800_46.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/1c0b1cbd-4917-44b4-b859-d2b9059cf389_calibrate_898800_46.stderr parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/5b5c2708-e781-4ba7-b5af-d25ac262c191_characterizeImage_898800_46.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/5b5c2708-e781-4ba7-b5af-d25ac262c191_characterizeImage_898800_46.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c20de635456102991cfd3c8542a7982 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/4ecf9520-6035-41cd-834d-fd4980285f17_calibrate_898694_48.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/4ecf9520-6035-41cd-834d-fd4980285f17_calibrate_898694_48.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58440635ae9edfa672b21ffcf7ddb7f5 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash a4588dc4a22c0cbeee3ebeda0a46ee7a parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5cc180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 965 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/42127a0b-b93d-4a2b-9fac-13f7c71ee2cf_calibrate_898798_32.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/42127a0b-b93d-4a2b-9fac-13f7c71ee2cf_calibrate_898798_32.stderr parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/39702eb8-6ac7-42f5-9775-3fb7d7d1285e_characterizeImage_898798_32.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/39702eb8-6ac7-42f5-9775-3fb7d7d1285e_characterizeImage_898798_32.stderr parsl.dataflow.dflow INFO: Task 1069 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4aa323695e5a488d4aa492bbbaf08ccc with result from task 1069 parsl.dataflow.dflow DEBUG: Task 1069 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1070 has memoization hash 8023961f245a371fc498fa817bb75a7b parsl.dataflow.memoization INFO: Task 1070 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c329300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1070 try 0 launched on executor multi with executor id 966 parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/4ed9a6e6-6602-4707-b9b1-b74a36a9a105_calibrate_898696_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/4ed9a6e6-6602-4707-b9b1-b74a36a9a105_calibrate_898696_11.stderr parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/dcd867b3-0992-4a38-9783-66af55b68770_characterizeImage_898696_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/dcd867b3-0992-4a38-9783-66af55b68770_characterizeImage_898696_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:44 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7e7e5f10f04de4f25bf3e1f7040c058 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash b44d913423193da1a4e21c4061e37169 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d587d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 967 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/be21b31b-6fde-401c-bdcb-47b1f96a4392_calibrate_898799_49.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/be21b31b-6fde-401c-bdcb-47b1f96a4392_calibrate_898799_49.stderr parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/778fc770-03a0-46f7-ad76-7cb81fdda818_characterizeImage_898799_49.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/778fc770-03a0-46f7-ad76-7cb81fdda818_characterizeImage_898799_49.stderr parsl.dataflow.dflow INFO: Task 418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cbb9adf649bbfdab42fac0bb84f2b5b with result from task 418 parsl.dataflow.dflow DEBUG: Task 418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 419 has memoization hash 2a8dbbcf72b10d9908c52f25ab3f260c parsl.dataflow.memoization INFO: Task 419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4804a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 968 parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/dc07d470-3104-48d3-aafa-0230005e358c_calibrate_898800_35.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/dc07d470-3104-48d3-aafa-0230005e358c_calibrate_898800_35.stderr parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/a9d7e5f4-3e0e-4dc4-9d77-cee7195913e7_characterizeImage_898800_35.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/a9d7e5f4-3e0e-4dc4-9d77-cee7195913e7_characterizeImage_898800_35.stderr parsl.dataflow.dflow INFO: Task 868 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfee51f45ffca0a977bba5911e91f084 with result from task 868 parsl.dataflow.dflow DEBUG: Task 868 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 869 has memoization hash 28a062a8ba39bfee1f6efa496f898a82 parsl.dataflow.memoization INFO: Task 869 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c441d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 869 try 0 launched on executor multi with executor id 969 parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/82e86838-a3e6-4247-87a9-0953a745196b_calibrate_898694_11.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/82e86838-a3e6-4247-87a9-0953a745196b_calibrate_898694_11.stderr parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/89eb401e-aa63-4301-9602-42dba77dd18e_characterizeImage_898694_11.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/89eb401e-aa63-4301-9602-42dba77dd18e_characterizeImage_898694_11.stderr parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e033c1304f2bf9d3ed13dbfb23d5077c with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/1ba59178-de10-4657-a371-7468053437e9_calibrate_898694_50.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/1ba59178-de10-4657-a371-7468053437e9_calibrate_898694_50.stderr parsl.dataflow.dflow INFO: Task 715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 989f698b5c03f4b66aa01d92738a89f4 with result from task 715 parsl.dataflow.dflow DEBUG: Task 715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 716 has memoization hash 941f33860961c311f4faea7e98387c5b parsl.dataflow.memoization INFO: Task 716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c377ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 716 try 0 launched on executor multi with executor id 970 parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/9f133c28-4a07-4999-8fb4-67994fa3640e_calibrate_898695_5.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/9f133c28-4a07-4999-8fb4-67994fa3640e_calibrate_898695_5.stderr parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/f7bc02f4-aac1-46b3-9da4-841a6316922d_characterizeImage_898695_5.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/f7bc02f4-aac1-46b3-9da4-841a6316922d_characterizeImage_898695_5.stderr parsl.dataflow.dflow INFO: Task 1012 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ee6d6f080655d9047a0f02698e59752 with result from task 1012 parsl.dataflow.dflow DEBUG: Task 1012 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1013 has memoization hash 33979d3e6fec80f791593101d5a0b5ba parsl.dataflow.memoization INFO: Task 1013 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2db380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1013 try 0 launched on executor multi with executor id 971 parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/c7896c29-b892-473b-b967-04e48acab913_calibrate_898696_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/c7896c29-b892-473b-b967-04e48acab913_calibrate_898696_24.stderr parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/3720f3ce-b3d4-446f-8571-1c702294eab7_characterizeImage_898696_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/3720f3ce-b3d4-446f-8571-1c702294eab7_characterizeImage_898696_24.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4456776c1a7e7755bbf4492452e4c880 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/04d52b69-6d52-4a59-a61f-27e65a749906_calibrate_898695_48.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/04d52b69-6d52-4a59-a61f-27e65a749906_calibrate_898695_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:45 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd0252bb75c256be58e402cddc270257 with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash 6c318de4e443a66efc243935240ed35a parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 972 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/79814f94-a4bc-4678-acb5-97c4b052ad04_calibrate_898798_17.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/79814f94-a4bc-4678-acb5-97c4b052ad04_calibrate_898798_17.stderr parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/005478be-e3b3-4992-8f7a-555c13b81168_characterizeImage_898798_17.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/005478be-e3b3-4992-8f7a-555c13b81168_characterizeImage_898798_17.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14b8c398cd694627da09431175408f8b with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/25e2fc2e-6cc4-4683-a3be-0168dd7242f3_calibrate_898696_1.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/25e2fc2e-6cc4-4683-a3be-0168dd7242f3_calibrate_898696_1.stderr parsl.dataflow.dflow INFO: Task 1018 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06c5f2236cbe5688ce79473009023bba with result from task 1018 parsl.dataflow.dflow DEBUG: Task 1018 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1019 has memoization hash 180a92dfbc5dc0cb064f24fe3a7d37b1 parsl.dataflow.memoization INFO: Task 1019 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2db6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1019 try 0 launched on executor multi with executor id 973 parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/6888f02d-4624-4cd7-94db-d62b259d20bb_calibrate_898800_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/6888f02d-4624-4cd7-94db-d62b259d20bb_calibrate_898800_22.stderr parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/f24380c1-818a-4a62-bcbb-ce3d15491fb0_characterizeImage_898800_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/f24380c1-818a-4a62-bcbb-ce3d15491fb0_characterizeImage_898800_22.stderr parsl.dataflow.dflow INFO: Task 586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db73cd82ef2c749fab451944d18eebaf with result from task 586 parsl.dataflow.dflow DEBUG: Task 586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 587 has memoization hash a5d56c2ac32fdffb3832a8b90525a8e7 parsl.dataflow.memoization INFO: Task 587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4df060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 587 try 0 launched on executor multi with executor id 974 parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/1f6c643d-e470-492c-a047-ab8944a69dbe_calibrate_898799_29.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/1f6c643d-e470-492c-a047-ab8944a69dbe_calibrate_898799_29.stderr parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/17012c66-e273-4070-af33-c0c76f2fe7de_characterizeImage_898799_29.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/17012c66-e273-4070-af33-c0c76f2fe7de_characterizeImage_898799_29.stderr parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54cf608ffd961b625fcbf7dd8800c3f0 with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash 8b653a00884bcd82be18884cb1ea241a parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 975 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/c4ca6878-008d-479a-ae80-243e30837d7e_calibrate_898799_25.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/c4ca6878-008d-479a-ae80-243e30837d7e_calibrate_898799_25.stderr parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/330a9d21-ee30-42c6-8c23-16ebc5ce9528_characterizeImage_898799_25.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/330a9d21-ee30-42c6-8c23-16ebc5ce9528_characterizeImage_898799_25.stderr parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 073d7ad39894d8a0e58d4c3e1ac32acf with result from task 406 parsl.dataflow.dflow DEBUG: Task 406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 407 has memoization hash fa6d122ba3ae916b25edc560410df878 parsl.dataflow.memoization INFO: Task 407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d663880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 976 parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/ca71a445-62c2-4a31-bfaf-3ab133f3dddc_calibrate_898800_25.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/ca71a445-62c2-4a31-bfaf-3ab133f3dddc_calibrate_898800_25.stderr parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/21c9b60c-a0a4-4709-a8f0-f5de8397c2e9_characterizeImage_898800_25.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/21c9b60c-a0a4-4709-a8f0-f5de8397c2e9_characterizeImage_898800_25.stderr parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac375c17effe10dca2266b8530606932 with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/77b16008-00f8-40a3-a09f-bac90807ffb3_calibrate_898695_58.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/77b16008-00f8-40a3-a09f-bac90807ffb3_calibrate_898695_58.stderr parsl.dataflow.dflow INFO: Task 694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a46556e6d33651853a9d4a93473d5b54 with result from task 694 parsl.dataflow.dflow DEBUG: Task 694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 695 has memoization hash 363a86b20c6f2c588d3eecaee0e6c98e parsl.dataflow.memoization INFO: Task 695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3765c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 695 try 0 launched on executor multi with executor id 977 parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/7a09d777-e987-44c8-bf2e-2c6cab429a1e_calibrate_898694_23.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/7a09d777-e987-44c8-bf2e-2c6cab429a1e_calibrate_898694_23.stderr parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/7da9b342-a8e6-40bd-a79f-cd181a9bd133_characterizeImage_898694_23.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/7da9b342-a8e6-40bd-a79f-cd181a9bd133_characterizeImage_898694_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:46 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 946 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bac2038a4344c51455696d63aa7c94d with result from task 946 parsl.dataflow.dflow DEBUG: Task 946 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 947 has memoization hash df4dc7fa77e9fe6cd034d3acd056afe3 parsl.dataflow.memoization INFO: Task 947 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2925c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 947 try 0 launched on executor multi with executor id 978 parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/10fa57a1-0bea-4738-9505-fcef5454b262_calibrate_898799_9.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/10fa57a1-0bea-4738-9505-fcef5454b262_calibrate_898799_9.stderr parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/a61146b2-edf4-4c83-b83c-f41607c7fd51_characterizeImage_898799_9.stdout parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/a61146b2-edf4-4c83-b83c-f41607c7fd51_characterizeImage_898799_9.stderr parsl.dataflow.dflow INFO: Task 724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66e43f2d363b4661723bf3964f3b42b3 with result from task 724 parsl.dataflow.dflow DEBUG: Task 724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 725 has memoization hash 18c053846991af12ecc75e8316615e62 parsl.dataflow.memoization INFO: Task 725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 725 try 0 launched on executor multi with executor id 979 parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/68f983a3-2ec0-46aa-a322-f164bd3bf10d_calibrate_898694_35.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/68f983a3-2ec0-46aa-a322-f164bd3bf10d_calibrate_898694_35.stderr parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/4af02fa4-cbbb-437b-b859-e6fa7615728f_characterizeImage_898694_35.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/4af02fa4-cbbb-437b-b859-e6fa7615728f_characterizeImage_898694_35.stderr parsl.dataflow.dflow INFO: Task 533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0df90aad7be31b1c901826edd3109102 with result from task 533 parsl.dataflow.dflow DEBUG: Task 533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/d95b67b1-6d86-4a04-8a59-839315d8e6f5_calibrate_898695_31.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/d95b67b1-6d86-4a04-8a59-839315d8e6f5_calibrate_898695_31.stderr parsl.dataflow.dflow INFO: Task 467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de266c90ca88256a6c7428f5d58ca304 with result from task 467 parsl.dataflow.dflow DEBUG: Task 467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/451144f1-db7f-438b-b0ee-9914d5f7f7d6_calibrate_898695_10.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/451144f1-db7f-438b-b0ee-9914d5f7f7d6_calibrate_898695_10.stderr parsl.dataflow.dflow INFO: Task 1021 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d331d57abc872f7d025c163853d3c1fe with result from task 1021 parsl.dataflow.dflow DEBUG: Task 1021 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1022 has memoization hash a4841ce4d6d45d8be344c9ec419dd7f0 parsl.dataflow.memoization INFO: Task 1022 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2dbb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1022 try 0 launched on executor multi with executor id 980 parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/b136bfd4-8230-4a28-a139-1404d176f03e_calibrate_898695_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/b136bfd4-8230-4a28-a139-1404d176f03e_calibrate_898695_33.stderr parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/9177541f-17ea-4580-accd-2074ad1b63a3_characterizeImage_898695_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/9177541f-17ea-4580-accd-2074ad1b63a3_characterizeImage_898695_33.stderr parsl.dataflow.dflow INFO: Task 739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cc23f314fa316fe15f411e407327730 with result from task 739 parsl.dataflow.dflow DEBUG: Task 739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 740 has memoization hash 924075db6b2ef22f95a4bc3711303167 parsl.dataflow.memoization INFO: Task 740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 740 try 0 launched on executor multi with executor id 981 parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/07c62024-7d8e-40ac-951f-dd5bb06ea7f3_calibrate_898694_49.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/07c62024-7d8e-40ac-951f-dd5bb06ea7f3_calibrate_898694_49.stderr parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/f172eb2e-c15b-45f6-a1ec-e8cabbc9015d_characterizeImage_898694_49.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/f172eb2e-c15b-45f6-a1ec-e8cabbc9015d_characterizeImage_898694_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:48 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 889 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8627b4fa4fe0cf9f691f3db6172a6f3 with result from task 889 parsl.dataflow.dflow DEBUG: Task 889 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 890 has memoization hash 6413df4da395c56bd59a9c16f16b8c46 parsl.dataflow.memoization INFO: Task 890 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4432e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 890 try 0 launched on executor multi with executor id 982 parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/07841f10-d3d6-410c-a4f9-af249370600f_calibrate_898694_52.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/07841f10-d3d6-410c-a4f9-af249370600f_calibrate_898694_52.stderr parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/8ff4aed3-5f64-4427-8e0e-a2449b816468_characterizeImage_898694_52.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/8ff4aed3-5f64-4427-8e0e-a2449b816468_characterizeImage_898694_52.stderr parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6657f0ac257a6785d6838440576f4987 with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash 9dfe100dacb8d85249fb523a9f1cbd51 parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d609c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 983 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/dffc3535-62ab-4ef3-af36-ff45996559f3_calibrate_898695_29.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/dffc3535-62ab-4ef3-af36-ff45996559f3_calibrate_898695_29.stderr parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/49ea2301-66ce-4f1f-9905-e2346711a880_characterizeImage_898695_29.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/49ea2301-66ce-4f1f-9905-e2346711a880_characterizeImage_898695_29.stderr parsl.dataflow.dflow INFO: Task 608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8e8f9d3c1723b94c9afc197931fc6ef with result from task 608 parsl.dataflow.dflow DEBUG: Task 608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/77cbf1b1-54a5-4da0-8681-03aad7e9b23e_calibrate_898696_27.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/77cbf1b1-54a5-4da0-8681-03aad7e9b23e_calibrate_898696_27.stderr parsl.dataflow.dflow INFO: Task 811 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23c257baa6422c1a662a7c1d54d5f177 with result from task 811 parsl.dataflow.dflow DEBUG: Task 811 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 812 has memoization hash f27fa66e30dcf1718c5c530620b1cfb0 parsl.dataflow.memoization INFO: Task 812 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 812 try 0 launched on executor multi with executor id 984 parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/22c4845a-52b5-4030-830e-838675df6f02_calibrate_898799_17.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/22c4845a-52b5-4030-830e-838675df6f02_calibrate_898799_17.stderr parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/8541934e-01ba-4fba-a5fd-0ad3ff8f4808_characterizeImage_898799_17.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/8541934e-01ba-4fba-a5fd-0ad3ff8f4808_characterizeImage_898799_17.stderr parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70ade959be19df2ce339d2eebddb4d41 with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/39c85a28-5a6f-423d-96d2-e8b6156755ca_calibrate_898800_57.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/39c85a28-5a6f-423d-96d2-e8b6156755ca_calibrate_898800_57.stderr parsl.dataflow.dflow INFO: Task 866 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7ad7eef27babb89a6036b85d853b31b with result from task 866 parsl.dataflow.dflow DEBUG: Task 866 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/36cd6351-d8f1-4f8b-9c37-4697f068f819_calibrate_898694_30.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/36cd6351-d8f1-4f8b-9c37-4697f068f819_calibrate_898694_30.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e066cbbd71f9cb2493d9f8e66fe1958 with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 735ceba1660b520539875da40fc9dd22 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ab920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 985 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/3eecc118-d7b5-4a0c-be0e-9d752d60ed64_calibrate_898695_60.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/3eecc118-d7b5-4a0c-be0e-9d752d60ed64_calibrate_898695_60.stderr parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/24f2afbb-0e97-4c00-b6c0-fc099f1a5f70_characterizeImage_898695_60.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/24f2afbb-0e97-4c00-b6c0-fc099f1a5f70_characterizeImage_898695_60.stderr parsl.dataflow.dflow INFO: Task 553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e787cdc4bf3a5060c3656b362a80215 with result from task 553 parsl.dataflow.dflow DEBUG: Task 553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 554 has memoization hash 91c82b481032f8f66dd610e10165a358 parsl.dataflow.memoization INFO: Task 554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dd1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 554 try 0 launched on executor multi with executor id 986 parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/328cd7f8-4999-4320-b70c-ed5701b4bbe2_calibrate_898800_29.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/328cd7f8-4999-4320-b70c-ed5701b4bbe2_calibrate_898800_29.stderr parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/066dc6af-9d54-4ce6-a446-f4dc552b10b9_characterizeImage_898800_29.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/066dc6af-9d54-4ce6-a446-f4dc552b10b9_characterizeImage_898800_29.stderr parsl.dataflow.dflow INFO: Task 688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry caf0050e526f99fffabd33bb7f8bc415 with result from task 688 parsl.dataflow.dflow DEBUG: Task 688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 689 has memoization hash f57fa9eedb88facc543594c7bc53d164 parsl.dataflow.memoization INFO: Task 689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c375580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 689 try 0 launched on executor multi with executor id 987 parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/8b131649-f1ab-413a-a521-760198806f61_calibrate_898798_29.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/8b131649-f1ab-413a-a521-760198806f61_calibrate_898798_29.stderr parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/df78ca11-275f-4dfb-907c-70332c874ff0_characterizeImage_898798_29.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/df78ca11-275f-4dfb-907c-70332c874ff0_characterizeImage_898798_29.stderr parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8265c41aa4137cdcb4b6d672f8ffc5c8 with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/77d03706-989e-4cd6-a607-20a478d9fac2_calibrate_898694_54.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/77d03706-989e-4cd6-a607-20a478d9fac2_calibrate_898694_54.stderr parsl.dataflow.dflow INFO: Task 536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95c9df7e3467c9a45821c89d2151100e with result from task 536 parsl.dataflow.dflow DEBUG: Task 536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/67a34143-0a91-4d44-9a92-fbca4dc04eee_calibrate_898696_29.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/67a34143-0a91-4d44-9a92-fbca4dc04eee_calibrate_898696_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:49 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9253716e473646433c98bf9bc5e8e2f2 with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/cdc21c34-7dec-4f62-8dca-3aca29b7bb87_calibrate_898695_51.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/cdc21c34-7dec-4f62-8dca-3aca29b7bb87_calibrate_898695_51.stderr parsl.dataflow.dflow INFO: Task 445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8379fd5146b74ab9ce34d04e57767f6 with result from task 445 parsl.dataflow.dflow DEBUG: Task 445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 446 has memoization hash 0a15c16dd7b54435b74082aad18ca63c parsl.dataflow.memoization INFO: Task 446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 446 try 0 launched on executor multi with executor id 988 parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/bf9a525d-4ba7-43c7-b308-3dd1d938ace9_calibrate_898798_55.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/bf9a525d-4ba7-43c7-b308-3dd1d938ace9_calibrate_898798_55.stderr parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/07726702-b872-4f1e-8caa-f70abd574a66_characterizeImage_898798_55.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/07726702-b872-4f1e-8caa-f70abd574a66_characterizeImage_898798_55.stderr parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62656e040e037596f3c592f165e6598f with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/32b0ead4-705a-4191-b922-12f6a784a892_calibrate_898695_17.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/32b0ead4-705a-4191-b922-12f6a784a892_calibrate_898695_17.stderr parsl.dataflow.dflow INFO: Task 412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bc603f8bc67564026f1b756dfe2dd24 with result from task 412 parsl.dataflow.dflow DEBUG: Task 412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 413 has memoization hash 8c3beb8f33debb714d4fd6b62366ddc0 parsl.dataflow.memoization INFO: Task 413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d663ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 413 try 0 launched on executor multi with executor id 989 parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/177bd906-a474-4156-bbdf-4b07978939db_calibrate_898694_6.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/177bd906-a474-4156-bbdf-4b07978939db_calibrate_898694_6.stderr parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/89a05b45-4b42-4b4b-80ab-ea4cd55f6e7a_characterizeImage_898694_6.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/89a05b45-4b42-4b4b-80ab-ea4cd55f6e7a_characterizeImage_898694_6.stderr parsl.dataflow.dflow INFO: Task 928 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b0fddba10327a7b49f7235abfd0646f with result from task 928 parsl.dataflow.dflow DEBUG: Task 928 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 929 has memoization hash e9ec3bc735c12ca0752854d7825c3344 parsl.dataflow.memoization INFO: Task 929 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2937e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 929 try 0 launched on executor multi with executor id 990 parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/517ede20-ed68-4558-ae68-0d27e70df0f6_calibrate_898696_16.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/517ede20-ed68-4558-ae68-0d27e70df0f6_calibrate_898696_16.stderr parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/2e111535-a3d1-484f-8fe1-155b7def2e12_characterizeImage_898696_16.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/2e111535-a3d1-484f-8fe1-155b7def2e12_characterizeImage_898696_16.stderr parsl.dataflow.dflow INFO: Task 1102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdbfca88db65fc0f6abcccfdc3ada566 with result from task 1102 parsl.dataflow.dflow DEBUG: Task 1102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1103 has memoization hash 6c399e3d51252a3e861efb77e9974e07 parsl.dataflow.memoization INFO: Task 1103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3511c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1103 try 0 launched on executor multi with executor id 991 parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/80dfd7a1-620e-47f1-b12b-c56d0f78e28b_calibrate_898695_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/80dfd7a1-620e-47f1-b12b-c56d0f78e28b_calibrate_898695_3.stderr parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/5fd2d5e5-5cb0-4a54-87b3-16a8ebfca6fa_characterizeImage_898695_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/5fd2d5e5-5cb0-4a54-87b3-16a8ebfca6fa_characterizeImage_898695_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:50 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f78fd1aacb43db0185e6b5935915973d with result from task 577 parsl.dataflow.dflow DEBUG: Task 577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 578 has memoization hash 7556aa1912fa5b2f431d3b0e3b0f78c6 parsl.dataflow.memoization INFO: Task 578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4de480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 578 try 0 launched on executor multi with executor id 992 parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/cf1cba2e-43ca-45ba-965b-88d4c1a0f07c_calibrate_898695_16.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/cf1cba2e-43ca-45ba-965b-88d4c1a0f07c_calibrate_898695_16.stderr parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/bc783b00-8aac-41a9-9814-18c4a7999657_characterizeImage_898695_16.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/bc783b00-8aac-41a9-9814-18c4a7999657_characterizeImage_898695_16.stderr parsl.dataflow.dflow INFO: Task 730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f09bc5c5ceb60b5e068383360df69bb with result from task 730 parsl.dataflow.dflow DEBUG: Task 730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 731 has memoization hash 50fa1493679dc160276cfd800eb10747 parsl.dataflow.memoization INFO: Task 731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 731 try 0 launched on executor multi with executor id 993 parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/375b1dcd-c1ea-439b-84ee-2a97b8c3d192_calibrate_898696_50.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/375b1dcd-c1ea-439b-84ee-2a97b8c3d192_calibrate_898696_50.stderr parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/95f1fb3e-a964-472a-ab76-5d75ee2ec6e8_characterizeImage_898696_50.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/95f1fb3e-a964-472a-ab76-5d75ee2ec6e8_characterizeImage_898696_50.stderr parsl.dataflow.dflow INFO: Task 394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 678a1694eac1e4d7aea22b1744629179 with result from task 394 parsl.dataflow.dflow DEBUG: Task 394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 395 has memoization hash 327929318c9736360efc694f40cf2129 parsl.dataflow.memoization INFO: Task 395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d662980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 994 parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/df949289-307f-47da-9926-9f43a104f0fd_calibrate_898799_6.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/df949289-307f-47da-9926-9f43a104f0fd_calibrate_898799_6.stderr parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/86057240-0171-437d-8008-20f4b0fc5588_characterizeImage_898799_6.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/86057240-0171-437d-8008-20f4b0fc5588_characterizeImage_898799_6.stderr parsl.dataflow.dflow INFO: Task 712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2940499c8623fcc8707bb8b3fbda0e25 with result from task 712 parsl.dataflow.dflow DEBUG: Task 712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 713 has memoization hash 1f248efeaa5877dfe55409b8dcb7f6ec parsl.dataflow.memoization INFO: Task 713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3779c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 713 try 0 launched on executor multi with executor id 995 parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/1e8f1a12-8dba-4b2d-82d3-2735ec2cd507_calibrate_898798_5.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/1e8f1a12-8dba-4b2d-82d3-2735ec2cd507_calibrate_898798_5.stderr parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/1325d26e-86d1-45ca-acc8-a3fc81ae3e6f_characterizeImage_898798_5.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/1325d26e-86d1-45ca-acc8-a3fc81ae3e6f_characterizeImage_898798_5.stderr parsl.dataflow.dflow INFO: Task 983 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50613e85f53185aa7a85d72305d99e23 with result from task 983 parsl.dataflow.dflow DEBUG: Task 983 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/2af6388d-fd7f-47af-85cd-226d4e4c6c34_calibrate_898696_34.stdout parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/2af6388d-fd7f-47af-85cd-226d4e4c6c34_calibrate_898696_34.stderr parsl.dataflow.dflow INFO: Task 416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d811da4aca86e871d056b859621d9ee5 with result from task 416 parsl.dataflow.dflow DEBUG: Task 416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/45407b46-ecba-485a-aba9-80d0de53d9f8_calibrate_898695_46.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/45407b46-ecba-485a-aba9-80d0de53d9f8_calibrate_898695_46.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0204d2e3caa220a09f7e62701f95cca0 with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/9b54da11-a3b0-4f2d-9364-440f8c77abf0_calibrate_898696_41.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/9b54da11-a3b0-4f2d-9364-440f8c77abf0_calibrate_898696_41.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5b0aa74853c0619464fd38b0c354ed1 with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/93db3b8d-71fe-434a-b80f-acb432cdad03_calibrate_898695_25.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/93db3b8d-71fe-434a-b80f-acb432cdad03_calibrate_898695_25.stderr parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09b010cf0e36aa1eed04cbddb03f249d with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/9e3b741c-2923-4e7f-a128-b478e0ef6a0c_calibrate_898696_38.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/9e3b741c-2923-4e7f-a128-b478e0ef6a0c_calibrate_898696_38.stderr parsl.dataflow.dflow INFO: Task 622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 662e1e68b27bdf0d1de2d67a20cc45f5 with result from task 622 parsl.dataflow.dflow DEBUG: Task 622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 623 has memoization hash 4e20c24f5a0533c6039c735bb0f18123 parsl.dataflow.memoization INFO: Task 623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c511bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 623 try 0 launched on executor multi with executor id 996 parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/fa9f52c3-c230-46d4-a2fc-be4c4f85602f_calibrate_898696_4.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/fa9f52c3-c230-46d4-a2fc-be4c4f85602f_calibrate_898696_4.stderr parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/19ffa1ec-f41d-4f29-86d2-8381cb651c8c_characterizeImage_898696_4.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/19ffa1ec-f41d-4f29-86d2-8381cb651c8c_characterizeImage_898696_4.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1dda491908ea52945d6b233d707a661b with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/a225bc31-deb5-4bb2-a6b4-7c3e46e8beeb_calibrate_898696_18.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/a225bc31-deb5-4bb2-a6b4-7c3e46e8beeb_calibrate_898696_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:51 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 1001 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 515f0ef9e8ba556be5b6f3811deb20e1 with result from task 1001 parsl.dataflow.dflow DEBUG: Task 1001 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/e3fb9fc1-9d42-4db8-a913-c5204576f223_calibrate_898695_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/e3fb9fc1-9d42-4db8-a913-c5204576f223_calibrate_898695_42.stderr parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e05f1dcc8768e91a19cf0525c5e3d16 with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/716eb7c7-b003-47c9-8f11-ca0a533e2dd4_calibrate_898695_18.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/716eb7c7-b003-47c9-8f11-ca0a533e2dd4_calibrate_898695_18.stderr parsl.dataflow.dflow INFO: Task 568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d07b098be504e7b43198e67e70ffc27 with result from task 568 parsl.dataflow.dflow DEBUG: Task 568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 569 has memoization hash f6ad85185a1a41dfd919473bcf921f26 parsl.dataflow.memoization INFO: Task 569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c5102c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 569 try 0 launched on executor multi with executor id 997 parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/38cbf436-d110-471c-bf4d-8c04d4e37cc7_calibrate_898800_49.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/38cbf436-d110-471c-bf4d-8c04d4e37cc7_calibrate_898800_49.stderr parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/5c8a5a13-7669-4145-8ebc-35340c188e17_characterizeImage_898800_49.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/5c8a5a13-7669-4145-8ebc-35340c188e17_characterizeImage_898800_49.stderr parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05e4b5b5c4cb21cce26d8d7452dba4f0 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/93888075-38d6-4169-b6a6-255d28f9c246_calibrate_898800_7.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/93888075-38d6-4169-b6a6-255d28f9c246_calibrate_898800_7.stderr parsl.dataflow.dflow INFO: Task 749 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 511d3187a6d7179f2a5b4a6a1cb38246 with result from task 749 parsl.dataflow.dflow DEBUG: Task 749 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/72461243-3334-4ad4-bc69-93861e1a3053_calibrate_898696_62.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/72461243-3334-4ad4-bc69-93861e1a3053_calibrate_898696_62.stderr parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef2ecd6e52ac1b7cfa9da0579684da9e with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/cdf98ae9-1ed9-4b9d-b2db-94bfab1cd3a7_calibrate_898694_46.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/cdf98ae9-1ed9-4b9d-b2db-94bfab1cd3a7_calibrate_898694_46.stderr parsl.dataflow.dflow INFO: Task 728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 371d54f637f676ab93b63efc085ffbc1 with result from task 728 parsl.dataflow.dflow DEBUG: Task 728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/42f7c779-c4ce-45ac-9e7d-8855f618f8ae_calibrate_898695_57.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/42f7c779-c4ce-45ac-9e7d-8855f618f8ae_calibrate_898695_57.stderr parsl.dataflow.dflow INFO: Task 937 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdb6cf5ec2d6b82900be48dfe8f172ab with result from task 937 parsl.dataflow.dflow DEBUG: Task 937 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 938 has memoization hash 4db653bb5ba760d363d3e2b71f284697 parsl.dataflow.memoization INFO: Task 938 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c291bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 938 try 0 launched on executor multi with executor id 998 parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/f7b5104d-046c-47b0-a00f-ba1593daf00b_calibrate_898695_28.stdout parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/f7b5104d-046c-47b0-a00f-ba1593daf00b_calibrate_898695_28.stderr parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/253d9316-c469-4212-8d90-f6c1ca3c42fd_characterizeImage_898695_28.stdout parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/253d9316-c469-4212-8d90-f6c1ca3c42fd_characterizeImage_898695_28.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c12e160036598ff12fecd87aebd94d06 with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/58ce860f-940c-4f3d-9bf3-c17986ea3191_calibrate_898800_40.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/58ce860f-940c-4f3d-9bf3-c17986ea3191_calibrate_898800_40.stderr parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fb075befdc4b131cc1724dacd60ad2f with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/45da7648-702b-4c46-8d8e-6cc39fe8e8fa_calibrate_898694_3.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/45da7648-702b-4c46-8d8e-6cc39fe8e8fa_calibrate_898694_3.stderr parsl.dataflow.dflow INFO: Task 455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9375e9a95e8ddc1abaa1ec3bf49fdc99 with result from task 455 parsl.dataflow.dflow DEBUG: Task 455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/01ba1150-bde7-4069-a60c-67ca1472d3a6_calibrate_898696_48.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/01ba1150-bde7-4069-a60c-67ca1472d3a6_calibrate_898696_48.stderr parsl.dataflow.dflow INFO: Task 773 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff034a870caaf7870aecb2c14bd89dd0 with result from task 773 parsl.dataflow.dflow DEBUG: Task 773 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/df617131-49f6-42eb-a49c-d0ce8ac8efd7_calibrate_898695_7.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/df617131-49f6-42eb-a49c-d0ce8ac8efd7_calibrate_898695_7.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78de77abb2d3f2dec12829bff1cc0e1d with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/d855c6bd-e0c6-4f51-83ed-0fd911525ca7_calibrate_898799_10.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/d855c6bd-e0c6-4f51-83ed-0fd911525ca7_calibrate_898799_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:53 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3deb883ad37102bd292b01485d0a03f1 with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/69234db1-835a-4f65-bd43-fe502c9e482e_calibrate_898694_5.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/69234db1-835a-4f65-bd43-fe502c9e482e_calibrate_898694_5.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c53827cb375d858cf00705d7fd5ab5f with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 5002cbf74361ad86f88aae058c5ba5f0 parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5a8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 999 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/b35ce77d-fa8a-450b-860d-daf9f70a9ecd_calibrate_898694_55.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/b35ce77d-fa8a-450b-860d-daf9f70a9ecd_calibrate_898694_55.stderr parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/55352941-c719-46bf-a64f-86e03424aed1_characterizeImage_898694_55.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/55352941-c719-46bf-a64f-86e03424aed1_characterizeImage_898694_55.stderr parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a047d710faccf44546a0632ec5ebb88e with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash eea46792047603e13f83860f452f59ee parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ed440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 1000 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/794222b2-e11b-4dd8-a760-c746fac8f672_calibrate_898696_55.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/794222b2-e11b-4dd8-a760-c746fac8f672_calibrate_898696_55.stderr parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/6f7c3096-33bf-436b-98ea-851b72f648ba_characterizeImage_898696_55.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/6f7c3096-33bf-436b-98ea-851b72f648ba_characterizeImage_898696_55.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17308b273f4098280f5f430aac73ec7a with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/8a153b53-76b8-46e1-9177-66bc09a2412d_calibrate_898800_54.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/8a153b53-76b8-46e1-9177-66bc09a2412d_calibrate_898800_54.stderr parsl.dataflow.dflow INFO: Task 1007 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9afe1f5cec199e7e4c002b83ab4c1420 with result from task 1007 parsl.dataflow.dflow DEBUG: Task 1007 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/f3ae9237-0dd3-4017-a2b2-b6c3d541f6a8_calibrate_898695_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/f3ae9237-0dd3-4017-a2b2-b6c3d541f6a8_calibrate_898695_38.stderr parsl.dataflow.dflow INFO: Task 1085 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76ec9a6ccceff9c95590c04eadc918ca with result from task 1085 parsl.dataflow.dflow DEBUG: Task 1085 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/4e463847-5972-4270-905f-3a83caba0df4_calibrate_898695_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/4e463847-5972-4270-905f-3a83caba0df4_calibrate_898695_37.stderr parsl.dataflow.dflow INFO: Task 592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c407ea5eeb38e87070cec824193467e with result from task 592 parsl.dataflow.dflow DEBUG: Task 592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 593 has memoization hash a3c5fd385ebd7d3cd46d29c43f77967c parsl.dataflow.memoization INFO: Task 593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4df6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 593 try 0 launched on executor multi with executor id 1001 parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/7345660d-bd61-450f-80b7-4f3de3486cd9_calibrate_898798_25.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/7345660d-bd61-450f-80b7-4f3de3486cd9_calibrate_898798_25.stderr parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/c2c98fcb-ce66-4a4f-9d4a-c3db3a7d9ff3_characterizeImage_898798_25.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/c2c98fcb-ce66-4a4f-9d4a-c3db3a7d9ff3_characterizeImage_898798_25.stderr parsl.dataflow.dflow INFO: Task 515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f584ec7d3d7e83ff04cc018c61c69006 with result from task 515 parsl.dataflow.dflow DEBUG: Task 515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/50df710f-f03e-4478-b1d2-39a06b0154f2_calibrate_898696_53.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/50df710f-f03e-4478-b1d2-39a06b0154f2_calibrate_898696_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:54 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 806 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9deae41eaec089ec10d34d8e7d83474e with result from task 806 parsl.dataflow.dflow DEBUG: Task 806 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/f3536772-7870-4576-9c98-6f56a2d7af69_calibrate_898799_54.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/f3536772-7870-4576-9c98-6f56a2d7af69_calibrate_898799_54.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5092cbbb3cf79db9fe82400679103b74 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/8be4a658-ffbe-4bc8-a401-13cf130230b1_calibrate_898798_57.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/8be4a658-ffbe-4bc8-a401-13cf130230b1_calibrate_898798_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:55 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 1079 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb07ab0caafa9863a2e9259e145f5de9 with result from task 1079 parsl.dataflow.dflow DEBUG: Task 1079 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/25565815-0fe7-464d-9715-2b55b455551c_calibrate_898696_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/25565815-0fe7-464d-9715-2b55b455551c_calibrate_898696_58.stderr parsl.dataflow.dflow INFO: Task 625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55916d16015dfa032c752c5f6216e2ac with result from task 625 parsl.dataflow.dflow DEBUG: Task 625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 626 has memoization hash 73d1f2071bad4ecd319701d32ac4273c parsl.dataflow.memoization INFO: Task 626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c511ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 626 try 0 launched on executor multi with executor id 1002 parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/00208e56-f082-4665-ba5b-d43c89b8d4c6_calibrate_898694_28.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/00208e56-f082-4665-ba5b-d43c89b8d4c6_calibrate_898694_28.stderr parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/23368ced-211e-49af-89a5-4f4878049a38_characterizeImage_898694_28.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/23368ced-211e-49af-89a5-4f4878049a38_characterizeImage_898694_28.stderr parsl.dataflow.dflow INFO: Task 524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 095845982a337b72a6df93ee17554244 with result from task 524 parsl.dataflow.dflow DEBUG: Task 524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/05cefb02-a578-401b-8150-e140b399934c_calibrate_898696_10.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/05cefb02-a578-401b-8150-e140b399934c_calibrate_898696_10.stderr parsl.dataflow.dflow INFO: Task 872 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5851da1157bd08528b28aa72cda97119 with result from task 872 parsl.dataflow.dflow DEBUG: Task 872 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/b92560b7-c235-42ab-b3a3-275bc4c2a203_calibrate_898696_54.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/b92560b7-c235-42ab-b3a3-275bc4c2a203_calibrate_898696_54.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 305 try 0 failed parsl.dataflow.dflow INFO: Task 305 marked for retry parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/377242ec-5005-4e58-82c4-26bd207a9818_calibrate_898694_14.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/377242ec-5005-4e58-82c4-26bd207a9818_calibrate_898694_14.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash 813d931c046c9836b74a99f4f389670b parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 1 launched on executor multi with executor id 1003 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/377242ec-5005-4e58-82c4-26bd207a9818_calibrate_898694_14.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/377242ec-5005-4e58-82c4-26bd207a9818_calibrate_898694_14.stderr parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b11ea466329b3d034fdfe08682ae16d7 with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash d9ee7c9f2e7a230b2472350db4c9c9af parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 1004 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/2290919f-e4bf-4a7b-8fc2-646bbbc26258_calibrate_898694_25.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/2290919f-e4bf-4a7b-8fc2-646bbbc26258_calibrate_898694_25.stderr parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/16f58476-05f4-465f-81b2-2d2fc3a3d462_characterizeImage_898694_25.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/16f58476-05f4-465f-81b2-2d2fc3a3d462_characterizeImage_898694_25.stderr parsl.dataflow.dflow INFO: Task 763 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69d724c666e80fd3a500aa9f875e8ac6 with result from task 763 parsl.dataflow.dflow DEBUG: Task 763 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 764 has memoization hash 743850d95ab1218d0c8e881f600f6cd5 parsl.dataflow.memoization INFO: Task 764 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3baa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 764 try 0 launched on executor multi with executor id 1005 parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/705820fd-e08c-4cbd-b3d7-96fff3f681bd_calibrate_898798_49.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/705820fd-e08c-4cbd-b3d7-96fff3f681bd_calibrate_898798_49.stderr parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/635a5f6a-c7c9-465f-bfbf-22e5d2f43add_characterizeImage_898798_49.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/635a5f6a-c7c9-465f-bfbf-22e5d2f43add_characterizeImage_898798_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:57 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 823 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b29697de76ebe2bfa759556bfff11c03 with result from task 823 parsl.dataflow.dflow DEBUG: Task 823 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 824 has memoization hash df98eea0fead7f300e70067a3c1749c6 parsl.dataflow.memoization INFO: Task 824 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 824 try 0 launched on executor multi with executor id 1006 parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/8af5af83-38df-4683-8032-095ce0c2ae9f_calibrate_898800_17.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/8af5af83-38df-4683-8032-095ce0c2ae9f_calibrate_898800_17.stderr parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/0b491b7d-f470-47da-b801-1add5d03aaae_characterizeImage_898800_17.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/0b491b7d-f470-47da-b801-1add5d03aaae_characterizeImage_898800_17.stderr parsl.dataflow.dflow INFO: Task 781 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4df04e63a704bed3bf64235164cdded with result from task 781 parsl.dataflow.dflow DEBUG: Task 781 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 782 has memoization hash d137f99a14c7f08c6cabd8de46f218ed parsl.dataflow.memoization INFO: Task 782 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 782 try 0 launched on executor multi with executor id 1007 parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/1755b4e3-7489-4115-98a7-516fbb481437_calibrate_898800_28.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/1755b4e3-7489-4115-98a7-516fbb481437_calibrate_898800_28.stderr parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/f9decf9f-9926-4347-b72c-1d9e3d47aad7_characterizeImage_898800_28.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/f9decf9f-9926-4347-b72c-1d9e3d47aad7_characterizeImage_898800_28.stderr parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b01595cdde73f2d669272c99bdd45e61 with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/4301f74b-a89c-462e-b16c-49c2e6af82f0_calibrate_898694_1.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/4301f74b-a89c-462e-b16c-49c2e6af82f0_calibrate_898694_1.stderr parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70592edfdf3163ec66db34c699071322 with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/3586e44c-9e67-463b-acfe-b66dd66ce980_calibrate_898799_47.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/3586e44c-9e67-463b-acfe-b66dd66ce980_calibrate_898799_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:58 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12bd8f820466bb31a7c7fd6cd61854b0 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/257e67a7-46f5-42ef-89be-c3b94d53a720_calibrate_898800_36.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/257e67a7-46f5-42ef-89be-c3b94d53a720_calibrate_898800_36.stderr parsl.dataflow.dflow INFO: Task 617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48914a51d7f76b8228ef0f23adfd70c8 with result from task 617 parsl.dataflow.dflow DEBUG: Task 617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/dfa40ab7-a172-4536-9004-5461859ff661_calibrate_898695_22.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/dfa40ab7-a172-4536-9004-5461859ff661_calibrate_898695_22.stderr parsl.dataflow.dflow INFO: Task 967 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5df67e51a44b92b1b2f1a7c1be7c6c52 with result from task 967 parsl.dataflow.dflow DEBUG: Task 967 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 968 has memoization hash 7cb38879c440d7edb10abe1d4bfc57b6 parsl.dataflow.memoization INFO: Task 968 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 968 try 0 launched on executor multi with executor id 1008 parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/80e02ab6-9af0-43c8-bc04-b8094f8f4e59_calibrate_898695_36.stdout parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/80e02ab6-9af0-43c8-bc04-b8094f8f4e59_calibrate_898695_36.stderr parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/9e38a3e2-7ec2-4f69-8109-c563ab7bced0_characterizeImage_898695_36.stdout parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/9e38a3e2-7ec2-4f69-8109-c563ab7bced0_characterizeImage_898695_36.stderr parsl.dataflow.dflow INFO: Task 952 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1629ae4109e440dcd6c4a10290ed726a with result from task 952 parsl.dataflow.dflow DEBUG: Task 952 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 953 has memoization hash 248e60da8932eb6d81d61195d1af172c parsl.dataflow.memoization INFO: Task 953 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c292c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 953 try 0 launched on executor multi with executor id 1009 parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/65bc30d4-91d8-465b-bd8e-d208c112fdc2_calibrate_898798_16.stdout parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/65bc30d4-91d8-465b-bd8e-d208c112fdc2_calibrate_898798_16.stderr parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/96d9b086-7575-4b07-a154-a6c20680a85b_characterizeImage_898798_16.stdout parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/96d9b086-7575-4b07-a154-a6c20680a85b_characterizeImage_898798_16.stderr parsl.dataflow.dflow INFO: Task 955 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5459f04ef570b53e332c5262d9bf60a7 with result from task 955 parsl.dataflow.dflow DEBUG: Task 955 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 956 has memoization hash cf04e937a3427fe98700f21a9c68f79c parsl.dataflow.memoization INFO: Task 956 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c292ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 956 try 0 launched on executor multi with executor id 1010 parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/4fd6e5cb-c7e0-4b95-a610-d005ba44b67b_calibrate_898798_1.stdout parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/4fd6e5cb-c7e0-4b95-a610-d005ba44b67b_calibrate_898798_1.stderr parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ef6314c9-413b-4baa-8757-d43b37701062_characterizeImage_898798_1.stdout parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ef6314c9-413b-4baa-8757-d43b37701062_characterizeImage_898798_1.stderr parsl.dataflow.dflow INFO: Task 785 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fc34ecad3f61c92496f065eb9b69d1d with result from task 785 parsl.dataflow.dflow DEBUG: Task 785 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/634f47cd-bf7c-4d66-956d-a1072b322885_calibrate_898696_32.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/634f47cd-bf7c-4d66-956d-a1072b322885_calibrate_898696_32.stderr parsl.dataflow.dflow INFO: Task 653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3cb7b1cb61161f4cc4b9f0840c80bca1 with result from task 653 parsl.dataflow.dflow DEBUG: Task 653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/4c41eb90-e342-4e61-b835-6426f7d01235_calibrate_898696_40.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/4c41eb90-e342-4e61-b835-6426f7d01235_calibrate_898696_40.stderr parsl.dataflow.dflow INFO: Task 548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bdf42b7cf1c64982018beafe7976e83 with result from task 548 parsl.dataflow.dflow DEBUG: Task 548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/d888f731-a78b-4400-b9bd-104a3dd2bf94_calibrate_898696_37.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/d888f731-a78b-4400-b9bd-104a3dd2bf94_calibrate_898696_37.stderr parsl.dataflow.dflow INFO: Task 913 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea2fdfe774ca1a74d24125cd2cf1a2a7 with result from task 913 parsl.dataflow.dflow DEBUG: Task 913 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 914 has memoization hash 794bc8ba96dad960ab06a2fd489739de parsl.dataflow.memoization INFO: Task 914 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c290a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 914 try 0 launched on executor multi with executor id 1011 parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/3ff78c46-0340-4b04-9f2d-23a418a9441f_calibrate_898799_12.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/3ff78c46-0340-4b04-9f2d-23a418a9441f_calibrate_898799_12.stderr parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/c4972256-4b87-4038-94b3-dbe1eb4c41c4_characterizeImage_898799_12.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/c4972256-4b87-4038-94b3-dbe1eb4c41c4_characterizeImage_898799_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:48:59 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55bf0a37e3818847ba848ee01214ea69 with result from task 436 parsl.dataflow.dflow DEBUG: Task 436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 437 has memoization hash d0ed6a4f6d4e1cb8f00f130549e69fc9 parsl.dataflow.memoization INFO: Task 437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c481760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 1012 parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/b030deaa-e47d-4c51-8193-59016f884878_calibrate_898695_6.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/b030deaa-e47d-4c51-8193-59016f884878_calibrate_898695_6.stderr parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/20a14898-7ce5-4337-b438-9e3f712403ad_characterizeImage_898695_6.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/20a14898-7ce5-4337-b438-9e3f712403ad_characterizeImage_898695_6.stderr parsl.dataflow.dflow INFO: Task 566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea64ee82c6d7a87f049fb90ecac0d0ff with result from task 566 parsl.dataflow.dflow DEBUG: Task 566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/c8f64e27-16de-4a86-b664-8a64081be740_calibrate_898696_17.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/c8f64e27-16de-4a86-b664-8a64081be740_calibrate_898696_17.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02e61f810e09028ddb5ea5f2fd7a9ee5 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/086b15b4-5087-4884-bdc8-91ed27337297_calibrate_898799_24.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/086b15b4-5087-4884-bdc8-91ed27337297_calibrate_898799_24.stderr parsl.dataflow.dflow INFO: Task 733 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80e06e9050d4675ba51804a09b2e06e9 with result from task 733 parsl.dataflow.dflow DEBUG: Task 733 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 734 has memoization hash e827f8f2a67e597a7e8a6c3afe4dc8e1 parsl.dataflow.memoization INFO: Task 734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3b8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 734 try 0 launched on executor multi with executor id 1013 parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/353b9769-4f56-42d2-afa1-8ac1e4ab51c8_calibrate_898696_8.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/353b9769-4f56-42d2-afa1-8ac1e4ab51c8_calibrate_898696_8.stderr parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/c6efdefd-3dd1-4414-b4d6-ace0736c4137_characterizeImage_898696_8.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/c6efdefd-3dd1-4414-b4d6-ace0736c4137_characterizeImage_898696_8.stderr parsl.dataflow.dflow INFO: Task 751 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebdac79a2ca31b66f59c36b916c217ec with result from task 751 parsl.dataflow.dflow DEBUG: Task 751 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 752 has memoization hash 52493c34337ce01b709ec8c0a19c016a parsl.dataflow.memoization INFO: Task 752 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3ba160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 752 try 0 launched on executor multi with executor id 1014 parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/70d59114-f691-42f4-87a2-1c36f7110cba_calibrate_898800_43.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/70d59114-f691-42f4-87a2-1c36f7110cba_calibrate_898800_43.stderr parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/25b2c2d7-52e9-464b-a475-1e315326e128_characterizeImage_898800_43.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/25b2c2d7-52e9-464b-a475-1e315326e128_characterizeImage_898800_43.stderr parsl.dataflow.dflow INFO: Task 430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2414c8dd8732e329ecc1721b1a32e621 with result from task 430 parsl.dataflow.dflow DEBUG: Task 430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 431 has memoization hash ef67ae7a6b88d5b735e51b2ebd0fae4e parsl.dataflow.memoization INFO: Task 431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c480f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 431 try 0 launched on executor multi with executor id 1015 parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/44842707-2af9-4f33-abb3-3487b385dc52_calibrate_898799_33.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/44842707-2af9-4f33-abb3-3487b385dc52_calibrate_898799_33.stderr parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/532f5d16-2fa6-4d98-b93e-c8eafefca919_characterizeImage_898799_33.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/532f5d16-2fa6-4d98-b93e-c8eafefca919_characterizeImage_898799_33.stderr parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e15c3429c545ad1d7c33563d0faba8c with result from task 427 parsl.dataflow.dflow DEBUG: Task 427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 428 has memoization hash bc8119de51d8f0290dd24af938480c9c parsl.dataflow.memoization INFO: Task 428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c480c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 1016 parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/4496b301-6a96-4d87-b526-98a35a5fa0ca_calibrate_898799_19.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/4496b301-6a96-4d87-b526-98a35a5fa0ca_calibrate_898799_19.stderr parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/af7455e0-8179-41cc-b9aa-8d7b23eace84_characterizeImage_898799_19.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/af7455e0-8179-41cc-b9aa-8d7b23eace84_characterizeImage_898799_19.stderr parsl.dataflow.dflow INFO: Task 853 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 528f541be1f8cacb883fff78f2a50f63 with result from task 853 parsl.dataflow.dflow DEBUG: Task 853 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 854 has memoization hash e70f3ee4b8826b95a3dbd2146260ebd8 parsl.dataflow.memoization INFO: Task 854 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c440d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 854 try 0 launched on executor multi with executor id 1017 parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/540f691f-4209-4e06-9645-f78dab5edca9_calibrate_898694_8.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/540f691f-4209-4e06-9645-f78dab5edca9_calibrate_898694_8.stderr parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/be4d7d79-191a-439b-a951-bbc440d0280f_characterizeImage_898694_8.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/be4d7d79-191a-439b-a951-bbc440d0280f_characterizeImage_898694_8.stderr parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16d81a6c22afdbc61bef9a7351cdef93 with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/402a1bdb-108a-406b-b988-c2ebcecf93d8_calibrate_898696_26.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/402a1bdb-108a-406b-b988-c2ebcecf93d8_calibrate_898696_26.stderr parsl.dataflow.dflow INFO: Task 707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efc27564e9a83610ca70659571353910 with result from task 707 parsl.dataflow.dflow DEBUG: Task 707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/670eb3c8-44bb-4ec2-9e3e-72a7488abaae_calibrate_898696_47.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/670eb3c8-44bb-4ec2-9e3e-72a7488abaae_calibrate_898696_47.stderr parsl.dataflow.dflow INFO: Task 860 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c085d3eafd673eae115b054892667dda with result from task 860 parsl.dataflow.dflow DEBUG: Task 860 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/076ce221-76fe-4523-8b31-0d6f6affe420_calibrate_898695_30.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/076ce221-76fe-4523-8b31-0d6f6affe420_calibrate_898695_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:01 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ce4651bd3cf31b6bf054c7a7016a678 with result from task 659 parsl.dataflow.dflow DEBUG: Task 659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/4777370d-acb7-45e3-8d29-fec3e2891b56_calibrate_898696_22.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/4777370d-acb7-45e3-8d29-fec3e2891b56_calibrate_898696_22.stderr parsl.dataflow.dflow INFO: Task 613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb33f33e57570615950e88d72ab24172 with result from task 613 parsl.dataflow.dflow DEBUG: Task 613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 614 has memoization hash cd4f31959490e07b744a86a1d4d0e6cf parsl.dataflow.memoization INFO: Task 614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c511260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 614 try 0 launched on executor multi with executor id 1018 parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/9c20547e-31a6-42ef-addf-9bf691544836_calibrate_898800_30.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/9c20547e-31a6-42ef-addf-9bf691544836_calibrate_898800_30.stderr parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/f1af8fb0-c19f-4db8-9e59-14ca589c948a_characterizeImage_898800_30.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/f1af8fb0-c19f-4db8-9e59-14ca589c948a_characterizeImage_898800_30.stderr parsl.dataflow.dflow INFO: Task 1090 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e22cd787c9e8b03ea6cf63bd27f909c with result from task 1090 parsl.dataflow.dflow DEBUG: Task 1090 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1091 has memoization hash 295d67a9fdf024c2433fb435d7d83f2f parsl.dataflow.memoization INFO: Task 1091 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c350540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1091 try 0 launched on executor multi with executor id 1019 parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/9b6c248c-925c-46ec-b79c-e959b02f8a31_calibrate_898696_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/9b6c248c-925c-46ec-b79c-e959b02f8a31_calibrate_898696_19.stderr parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/9de6b8ac-60ff-413f-b6d5-5ff4cbabe482_characterizeImage_898696_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/9de6b8ac-60ff-413f-b6d5-5ff4cbabe482_characterizeImage_898696_19.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fc16c2fd561111d2ad00cd32bc28988 with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/b8faf874-026c-4833-b1a8-628f0d710287_calibrate_898799_36.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/b8faf874-026c-4833-b1a8-628f0d710287_calibrate_898799_36.stderr parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7228ed8c4f0093183b42112819353669 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash b3088e3bd967f064981ded5bd80f25d3 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5aa020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 1020 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/d2eb003e-0dc4-4adf-bbfa-ae866070e406_calibrate_898800_1.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/d2eb003e-0dc4-4adf-bbfa-ae866070e406_calibrate_898800_1.stderr parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/6b8aad91-534e-49a5-b141-e3f3f8a5476a_characterizeImage_898800_1.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/6b8aad91-534e-49a5-b141-e3f3f8a5476a_characterizeImage_898800_1.stderr parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69cdb759d815a58732504629c5756ced with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/269c8fe2-fbf0-4a3c-8939-728692b548f4_calibrate_898694_60.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/269c8fe2-fbf0-4a3c-8939-728692b548f4_calibrate_898694_60.stderr parsl.dataflow.dflow INFO: Task 875 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a1b0efb8f1e8180a54734b4399d870c with result from task 875 parsl.dataflow.dflow DEBUG: Task 875 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/f2fcbf99-de8d-4ac9-8e60-d849391ebd14_calibrate_898694_37.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/f2fcbf99-de8d-4ac9-8e60-d849391ebd14_calibrate_898694_37.stderr parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f45774fa008e93025764dd47e2d2cfa with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/75816b91-9de4-4ef6-afcd-445b7939a547_calibrate_898694_15.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/75816b91-9de4-4ef6-afcd-445b7939a547_calibrate_898694_15.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5342e3b1ca9f7ffc5c1a622c9cd9f5a with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 8798a4cee6c13eb8e62627c86f6f470b parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5abc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 1021 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/6364a26d-39bf-4644-9119-3e5580e7684e_calibrate_898799_58.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/6364a26d-39bf-4644-9119-3e5580e7684e_calibrate_898799_58.stderr parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/1f595f02-9b38-431f-9017-8da0297bd727_characterizeImage_898799_58.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/1f595f02-9b38-431f-9017-8da0297bd727_characterizeImage_898799_58.stderr parsl.dataflow.dflow INFO: Task 743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec1ae208f2ef361e5a2a7a1ca2102083 with result from task 743 parsl.dataflow.dflow DEBUG: Task 743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/bfc95f0f-e3a1-4524-9dbc-748b03680c63_calibrate_898694_24.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/bfc95f0f-e3a1-4524-9dbc-748b03680c63_calibrate_898694_24.stderr parsl.dataflow.dflow INFO: Task 905 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f54cf6470881686cb45f3c6f767e95a0 with result from task 905 parsl.dataflow.dflow DEBUG: Task 905 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/44c9aac1-e91d-4eb3-b49f-0bc0f4d90f6f_calibrate_898694_59.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/44c9aac1-e91d-4eb3-b49f-0bc0f4d90f6f_calibrate_898694_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:02 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 877 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69f94db51b6a4eeabb666fc407ca08d2 with result from task 877 parsl.dataflow.dflow DEBUG: Task 877 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 878 has memoization hash 6608d008be45914b0b2c88e3fdf44617 parsl.dataflow.memoization INFO: Task 878 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c442660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 878 try 0 launched on executor multi with executor id 1022 parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/3b4e0445-6ca8-47d5-8cc4-bb54ffb82ccc_calibrate_898696_25.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/3b4e0445-6ca8-47d5-8cc4-bb54ffb82ccc_calibrate_898696_25.stderr parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/f73fcf90-b226-4427-83ef-0ee48a96341c_characterizeImage_898696_25.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/f73fcf90-b226-4427-83ef-0ee48a96341c_characterizeImage_898696_25.stderr parsl.dataflow.dflow INFO: Task 389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37c857d99abbe016a9161bd1cb84d4c8 with result from task 389 parsl.dataflow.dflow DEBUG: Task 389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/129a4036-131a-4c10-bb6b-d32c33380bb3_calibrate_898694_38.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/129a4036-131a-4c10-bb6b-d32c33380bb3_calibrate_898694_38.stderr parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 027ef5c42fe821817ec75486d682b0df with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/485fa54b-f5db-4b1d-b27c-21eb344a00bb_calibrate_898694_58.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/485fa54b-f5db-4b1d-b27c-21eb344a00bb_calibrate_898694_58.stderr parsl.dataflow.dflow INFO: Task 383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 365cad3295f57dd9b2d7ef43d1bbcdc7 with result from task 383 parsl.dataflow.dflow DEBUG: Task 383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/e71634c4-a3b4-4248-9afa-22294ed75443_calibrate_898800_59.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/e71634c4-a3b4-4248-9afa-22294ed75443_calibrate_898800_59.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d4592c5b24f9fa9c3f69558d25fff31 with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/b18dbaa1-f478-4fda-9ea3-61679c5ab540_calibrate_898695_45.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/b18dbaa1-f478-4fda-9ea3-61679c5ab540_calibrate_898695_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:03 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 793 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 427f57779a473800837c7632ef28ff90 with result from task 793 parsl.dataflow.dflow DEBUG: Task 793 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 794 has memoization hash 3b38e5a00df0a7a72bdfb375e5d5e0ba parsl.dataflow.memoization INFO: Task 794 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3bbc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 794 try 0 launched on executor multi with executor id 1023 parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/4fdf0481-6645-43fc-b4ed-5286b4e9678d_calibrate_898800_44.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/4fdf0481-6645-43fc-b4ed-5286b4e9678d_calibrate_898800_44.stderr parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/7718c854-b248-46d2-a245-db7394b3be5e_characterizeImage_898800_44.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/7718c854-b248-46d2-a245-db7394b3be5e_characterizeImage_898800_44.stderr parsl.dataflow.dflow INFO: Task 778 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca93ba524a21724505a8b9edefa7466b with result from task 778 parsl.dataflow.dflow DEBUG: Task 778 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 779 has memoization hash b32452c53097f058ad9910cb8f34a9a6 parsl.dataflow.memoization INFO: Task 779 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 779 try 0 launched on executor multi with executor id 1024 parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/dadc5136-9729-4a59-8897-547da1780bd3_calibrate_898800_13.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/dadc5136-9729-4a59-8897-547da1780bd3_calibrate_898800_13.stderr parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/778e0cd9-e824-499a-a43b-2f00a433a97f_characterizeImage_898800_13.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/778e0cd9-e824-499a-a43b-2f00a433a97f_characterizeImage_898800_13.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4b1a667e668c726892ae0ac6e4463c1 with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/f7ff8c72-b0c8-4985-a73a-658c4c8ecb8e_calibrate_898800_24.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/f7ff8c72-b0c8-4985-a73a-658c4c8ecb8e_calibrate_898800_24.stderr parsl.dataflow.dflow INFO: Task 838 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e93c0721e26383db889a626d3113b699 with result from task 838 parsl.dataflow.dflow DEBUG: Task 838 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 839 has memoization hash 2b216891f4f71de97dd4c4eb7359c666 parsl.dataflow.memoization INFO: Task 839 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 839 try 0 launched on executor multi with executor id 1025 parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/f572d6d7-7f23-4d74-bd66-bf8674cec333_calibrate_898798_44.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/f572d6d7-7f23-4d74-bd66-bf8674cec333_calibrate_898798_44.stderr parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/721711bc-4592-4e46-a01f-e7a0ac33c36a_characterizeImage_898798_44.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/721711bc-4592-4e46-a01f-e7a0ac33c36a_characterizeImage_898798_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:03 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d6bbc9bd39ea774535d51b005649bdc with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/9478d13b-30c4-4be1-a5f7-f81fd1dd2aad_calibrate_898696_13.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/9478d13b-30c4-4be1-a5f7-f81fd1dd2aad_calibrate_898696_13.stderr parsl.dataflow.dflow INFO: Task 605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ba56cdb911350dd74ef28deb65a0145 with result from task 605 parsl.dataflow.dflow DEBUG: Task 605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/097c61cd-a697-442b-973e-dcf72dd6983b_calibrate_898800_51.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/097c61cd-a697-442b-973e-dcf72dd6983b_calibrate_898800_51.stderr parsl.dataflow.dflow INFO: Task 931 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41acb0d6322d657a06bf4053830e6615 with result from task 931 parsl.dataflow.dflow DEBUG: Task 931 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 932 has memoization hash 80e113d5e863b2ca6d6a05828a5565d5 parsl.dataflow.memoization INFO: Task 932 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c291620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 932 try 0 launched on executor multi with executor id 1026 parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/2f904687-1650-4236-8999-3554b52dc5cd_calibrate_898694_9.stdout parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/2f904687-1650-4236-8999-3554b52dc5cd_calibrate_898694_9.stderr parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/fc370d9a-1c80-4e82-9479-05c4e817ee26_characterizeImage_898694_9.stdout parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/fc370d9a-1c80-4e82-9479-05c4e817ee26_characterizeImage_898694_9.stderr parsl.dataflow.dflow INFO: Task 499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24b3544f3910716e867e86c5546a070d with result from task 499 parsl.dataflow.dflow DEBUG: Task 499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 500 has memoization hash 6026b06d6c309cea767989e82e3503ae parsl.dataflow.memoization INFO: Task 500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 500 try 0 launched on executor multi with executor id 1027 parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/bb62a931-b47d-4408-89b1-61ff44a856e1_calibrate_898799_30.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/bb62a931-b47d-4408-89b1-61ff44a856e1_calibrate_898799_30.stderr parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/44f4b2af-92c4-4126-b94f-068da0e4f109_characterizeImage_898799_30.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/44f4b2af-92c4-4126-b94f-068da0e4f109_characterizeImage_898799_30.stderr parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9282eaedd3e8b9d5eb3e59082a84390 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/f4952490-d9f6-47cd-9093-5f1e74b5d43c_calibrate_898800_27.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/f4952490-d9f6-47cd-9093-5f1e74b5d43c_calibrate_898800_27.stderr parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eab6ea866cade56c2bd74e18793d9845 with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/c19ec2df-8468-4b72-a23e-808108cf6f0d_calibrate_898800_15.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/c19ec2df-8468-4b72-a23e-808108cf6f0d_calibrate_898800_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:04 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe2947ffa00d6e1fbde3a80299677b8c with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash a4e2ac7d4d68fed80f3892b6e007b42f parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ed760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 1028 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/25fa34c1-4166-4fe3-a2c5-e5180e360a6e_calibrate_898799_37.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/25fa34c1-4166-4fe3-a2c5-e5180e360a6e_calibrate_898799_37.stderr parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/88f83bee-26c0-4188-9562-ef260a2ca606_characterizeImage_898799_37.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/88f83bee-26c0-4188-9562-ef260a2ca606_characterizeImage_898799_37.stderr parsl.dataflow.dflow INFO: Task 560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 876fc71505db461da977284d74bd5a85 with result from task 560 parsl.dataflow.dflow DEBUG: Task 560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/4c0458a3-9709-4d33-958b-f1f92ad60840_calibrate_898695_59.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/4c0458a3-9709-4d33-958b-f1f92ad60840_calibrate_898695_59.stderr parsl.dataflow.dflow INFO: Task 737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68b3fd120edaf3ac2efdf05f8ddaee9e with result from task 737 parsl.dataflow.dflow DEBUG: Task 737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/c26a49d5-9855-437b-9d11-d3f4685fa3dc_calibrate_898696_46.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/c26a49d5-9855-437b-9d11-d3f4685fa3dc_calibrate_898696_46.stderr parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c2bb2f0a43e8dee9513dd183f6b2c34 with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/5ebab2f8-b8be-4192-8a00-1e8d50ec3465_calibrate_898798_52.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/5ebab2f8-b8be-4192-8a00-1e8d50ec3465_calibrate_898798_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:05 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 907 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9beefb19752e7dfd64d50e8c5635c82 with result from task 907 parsl.dataflow.dflow DEBUG: Task 907 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 908 has memoization hash eab6758fb96c928219c663d5db9d34ce parsl.dataflow.memoization INFO: Task 908 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c290400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 908 try 0 launched on executor multi with executor id 1029 parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/1300a700-e5b4-447a-aecb-49e189df15df_calibrate_898696_3.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/1300a700-e5b4-447a-aecb-49e189df15df_calibrate_898696_3.stderr parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/44dc3ea1-a77a-49d3-ac50-ac69c2b47ab6_characterizeImage_898696_3.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/44dc3ea1-a77a-49d3-ac50-ac69c2b47ab6_characterizeImage_898696_3.stderr parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92edf577eb80fb925f662554cef03695 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/293243b8-d4c4-4189-bb32-49dd7dcea659_calibrate_898696_52.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/293243b8-d4c4-4189-bb32-49dd7dcea659_calibrate_898696_52.stderr parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee92661a16c310535ac25f5031de31e7 with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/95f04615-b52b-4ebc-b505-b2cf4516cd8b_calibrate_898799_52.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/95f04615-b52b-4ebc-b505-b2cf4516cd8b_calibrate_898799_52.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4375ee0f2cf507ea3963886d919513a8 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/9e15c895-4628-44bb-9433-c8814624c1a9_calibrate_898694_36.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/9e15c895-4628-44bb-9433-c8814624c1a9_calibrate_898694_36.stderr parsl.dataflow.dflow INFO: Task 473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9362e675d52d92d9c0b9bb113d89e7c9 with result from task 473 parsl.dataflow.dflow DEBUG: Task 473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/5f289302-85d2-4532-af71-d7a3f9209595_calibrate_898695_14.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/5f289302-85d2-4532-af71-d7a3f9209595_calibrate_898695_14.stderr parsl.dataflow.dflow INFO: Task 460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0c312e47bb02479fe6f6e20d1982f7a with result from task 460 parsl.dataflow.dflow DEBUG: Task 460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 461 has memoization hash e95f900783800f781da24f5e5a3fcc98 parsl.dataflow.memoization INFO: Task 461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 1030 parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/2d17d49a-44d3-47d4-b792-8229ca2176de_calibrate_898695_11.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/2d17d49a-44d3-47d4-b792-8229ca2176de_calibrate_898695_11.stderr parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/637a89f5-32e4-4123-a566-533fa29846ec_characterizeImage_898695_11.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/637a89f5-32e4-4123-a566-533fa29846ec_characterizeImage_898695_11.stderr parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db7179dfb22c8b75c302ef687ebcf361 with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/9ea68e1d-b4e6-409a-a172-5486a0efce9a_calibrate_898694_41.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/9ea68e1d-b4e6-409a-a172-5486a0efce9a_calibrate_898694_41.stderr parsl.dataflow.dflow INFO: Task 925 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59e225bcfd2056f6b822cda7c4248559 with result from task 925 parsl.dataflow.dflow DEBUG: Task 925 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 926 has memoization hash 5874f13a53b1a8e303c1ebcb013d5b6b parsl.dataflow.memoization INFO: Task 926 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c293b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 926 try 0 launched on executor multi with executor id 1031 parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/79f5e065-ac44-4bc8-87e2-8abfac2317c8_calibrate_898695_50.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/79f5e065-ac44-4bc8-87e2-8abfac2317c8_calibrate_898695_50.stderr parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/70f44ad4-c1ac-460c-bbfe-efd4868d8f0a_characterizeImage_898695_50.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/70f44ad4-c1ac-460c-bbfe-efd4868d8f0a_characterizeImage_898695_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:05 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 1072 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2dc9182bdb0620a8bc03c2079467b50 with result from task 1072 parsl.dataflow.dflow DEBUG: Task 1072 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1073 has memoization hash e68dffca902b875f2d6da9518a25b042 parsl.dataflow.memoization INFO: Task 1073 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c329580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1073 try 0 launched on executor multi with executor id 1032 parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/940e8366-0bff-4953-8982-e6472d9d702e_calibrate_898799_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/940e8366-0bff-4953-8982-e6472d9d702e_calibrate_898799_43.stderr parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/3457e19d-e614-4fff-9940-e110e69df94f_characterizeImage_898799_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/3457e19d-e614-4fff-9940-e110e69df94f_characterizeImage_898799_43.stderr parsl.dataflow.dflow INFO: Task 1094 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 978f6504e0ae94da82a75940bbf7911f with result from task 1094 parsl.dataflow.dflow DEBUG: Task 1094 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/392ca369-d0cd-446b-aed2-2dba357e4d1d_calibrate_898694_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/392ca369-d0cd-446b-aed2-2dba357e4d1d_calibrate_898694_12.stderr parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 867a43e62a422db1171e0b8bf03f0dc7 with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/08d52931-8a23-49d2-8887-c0f474a6f107_calibrate_898800_12.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/08d52931-8a23-49d2-8887-c0f474a6f107_calibrate_898800_12.stderr parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f408570ac0543da2f52e963e649747f4 with result from task 398 parsl.dataflow.dflow DEBUG: Task 398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/0ebae809-8f3e-455d-b7d4-c4bcdc9d78d1_calibrate_898694_7.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/0ebae809-8f3e-455d-b7d4-c4bcdc9d78d1_calibrate_898694_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:06 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4e3f98b4a1a349537d62dcd6ed30404 with result from task 538 parsl.dataflow.dflow DEBUG: Task 538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 539 has memoization hash 050cef99d055545deae8e5edf1d05a23 parsl.dataflow.memoization INFO: Task 539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dc400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 539 try 0 launched on executor multi with executor id 1033 parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/75e066f3-cb9a-4348-b558-62fbb8c110c9_calibrate_898800_32.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/75e066f3-cb9a-4348-b558-62fbb8c110c9_calibrate_898800_32.stderr parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/13a56d53-c439-4415-b6c4-8b2848a05ff6_characterizeImage_898800_32.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/13a56d53-c439-4415-b6c4-8b2848a05ff6_characterizeImage_898800_32.stderr parsl.dataflow.dflow INFO: Task 440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8efd19bf852fa843fb4f84a1378aa1e with result from task 440 parsl.dataflow.dflow DEBUG: Task 440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/49fef6c9-bc54-45ab-a3ba-415bb62d5f15_calibrate_898800_47.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/49fef6c9-bc54-45ab-a3ba-415bb62d5f15_calibrate_898800_47.stderr parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 595cca24b6eaa46a295b2785d193a262 with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/856ea1ad-e8d9-4f15-8975-64aeab8e2952_calibrate_898696_59.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/856ea1ad-e8d9-4f15-8975-64aeab8e2952_calibrate_898696_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:07 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b30ca7ee3c1ea56b99d8bc84d0453cc1 with result from task 496 parsl.dataflow.dflow DEBUG: Task 496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 497 has memoization hash a016403407af9bed60019175f9e3c977 parsl.dataflow.memoization INFO: Task 497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c483c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 497 try 0 launched on executor multi with executor id 1034 parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/dfc41065-6e5d-4754-b513-eb1b05a9713b_calibrate_898695_8.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/dfc41065-6e5d-4754-b513-eb1b05a9713b_calibrate_898695_8.stderr parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/e9b3f818-f495-4062-8a52-442a54674f59_characterizeImage_898695_8.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/e9b3f818-f495-4062-8a52-442a54674f59_characterizeImage_898695_8.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9ab423a8ea28930aada304b92b51bd2 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/46da9d85-c5bf-4321-9280-8ef233ccdccc_calibrate_898695_44.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/46da9d85-c5bf-4321-9280-8ef233ccdccc_calibrate_898695_44.stderr parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b05b9a13af81bb2c244eb6b67459e244 with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/8bffb5ee-161d-4d60-aa3d-15894630a60e_calibrate_898799_23.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/8bffb5ee-161d-4d60-aa3d-15894630a60e_calibrate_898799_23.stderr parsl.dataflow.dflow INFO: Task 679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80e19141a38bbc3c347048a5d599d1fc with result from task 679 parsl.dataflow.dflow DEBUG: Task 679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 680 has memoization hash 581bdfa90b2f6e9adfbc9da78cfbf99a parsl.dataflow.memoization INFO: Task 680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c374ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 680 try 0 launched on executor multi with executor id 1035 parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/76c675ff-d57e-46a8-ade1-83abc1ccfb98_calibrate_898696_44.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/76c675ff-d57e-46a8-ade1-83abc1ccfb98_calibrate_898696_44.stderr parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/a1ecec3d-6d4b-41da-8896-db74a4c30379_characterizeImage_898696_44.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/a1ecec3d-6d4b-41da-8896-db74a4c30379_characterizeImage_898696_44.stderr parsl.dataflow.dflow INFO: Task 980 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d05b90e53297be3c3b28b18e2ee713a9 with result from task 980 parsl.dataflow.dflow DEBUG: Task 980 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/c1e96afe-ae4e-4941-b2ad-d7e802f0b5de_calibrate_898800_21.stdout parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/c1e96afe-ae4e-4941-b2ad-d7e802f0b5de_calibrate_898800_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:07 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 867d4493385f02c14877c41ddd3f7238 with result from task 464 parsl.dataflow.dflow DEBUG: Task 464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/f7eb1141-9965-4791-999d-f615d2a9b0bb_calibrate_898798_56.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/f7eb1141-9965-4791-999d-f615d2a9b0bb_calibrate_898798_56.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e98a86d25497cd0f2925447d17e749e with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash b24c45cefb2392d628dc6a519e69873e parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d608680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 1036 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/8b476ca5-2c03-479d-a372-df1adce0b6c5_calibrate_898695_39.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/8b476ca5-2c03-479d-a372-df1adce0b6c5_calibrate_898695_39.stderr parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/738671f4-71d6-440e-b76d-d0b61b151e85_characterizeImage_898695_39.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/738671f4-71d6-440e-b76d-d0b61b151e85_characterizeImage_898695_39.stderr parsl.dataflow.dflow INFO: Task 848 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 626b766b86fbba0a1047c5511921cac3 with result from task 848 parsl.dataflow.dflow DEBUG: Task 848 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/4b0a708e-75c4-4977-bff3-7185722124c3_calibrate_898695_52.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/4b0a708e-75c4-4977-bff3-7185722124c3_calibrate_898695_52.stderr parsl.dataflow.dflow INFO: Task 488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50f69a7a97754aad8e679c65a0b60a89 with result from task 488 parsl.dataflow.dflow DEBUG: Task 488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/91aac08c-c5f3-4ec6-aad5-f40007aa5700_calibrate_898694_27.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/91aac08c-c5f3-4ec6-aad5-f40007aa5700_calibrate_898694_27.stderr parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f977d08a926edb85ecaf0c5a4364a213 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/2041cb76-6180-45d4-9ef3-91c6547c9ab9_calibrate_898798_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/2041cb76-6180-45d4-9ef3-91c6547c9ab9_calibrate_898798_15.stderr parsl.dataflow.dflow INFO: Task 857 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49a23227f1c192ef6d12a2b450a25e26 with result from task 857 parsl.dataflow.dflow DEBUG: Task 857 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/0032e136-30f0-4506-a459-f2ec1fe3d711_calibrate_898694_51.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/0032e136-30f0-4506-a459-f2ec1fe3d711_calibrate_898694_51.stderr parsl.dataflow.dflow INFO: Task 595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa6d8b98fb270d176210c2b4f8e9aea3 with result from task 595 parsl.dataflow.dflow DEBUG: Task 595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 596 has memoization hash 46414ece9bda60b78a8f5a9b2cfd27d1 parsl.dataflow.memoization INFO: Task 596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4df740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 596 try 0 launched on executor multi with executor id 1037 parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/f672bb83-5a6d-4f52-8c3e-964482e9bfec_calibrate_898694_45.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/f672bb83-5a6d-4f52-8c3e-964482e9bfec_calibrate_898694_45.stderr parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/6dcb40de-2331-4005-b84f-5183df3feafb_characterizeImage_898694_45.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/6dcb40de-2331-4005-b84f-5183df3feafb_characterizeImage_898694_45.stderr parsl.dataflow.dflow INFO: Task 482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5076ae32905a210b7b375fe9a967e6df with result from task 482 parsl.dataflow.dflow DEBUG: Task 482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/3372209f-37c4-45aa-aa61-3a0ca245cc2e_calibrate_898799_59.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/3372209f-37c4-45aa-aa61-3a0ca245cc2e_calibrate_898799_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:08 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 767 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b240053786cc7ba4f61c5f1331c125c0 with result from task 767 parsl.dataflow.dflow DEBUG: Task 767 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/f57f0dc2-d049-4398-b58a-cb4a6e16a651_calibrate_898695_32.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/f57f0dc2-d049-4398-b58a-cb4a6e16a651_calibrate_898695_32.stderr parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0cd142090f1d2b63b8c7027eaeaca68 with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/2133ade3-9813-4daf-b0be-7a3221a65e69_calibrate_898799_22.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/2133ade3-9813-4daf-b0be-7a3221a65e69_calibrate_898799_22.stderr parsl.dataflow.dflow INFO: Task 668 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61f539d1e6e3519a9dd371a0bd0780f2 with result from task 668 parsl.dataflow.dflow DEBUG: Task 668 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/ca1954fe-6137-47d8-ad1b-0c3212daa9f0_calibrate_898800_11.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/ca1954fe-6137-47d8-ad1b-0c3212daa9f0_calibrate_898800_11.stderr parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04e4e9e9c4c0412e7c1fa4100dc58a67 with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash d11467cb47cd42a26a7f9dfce426b2ee parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6600e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 1038 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/26ae9a68-c5a5-4f67-ae7e-4b9cbfd47a90_calibrate_898695_12.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/26ae9a68-c5a5-4f67-ae7e-4b9cbfd47a90_calibrate_898695_12.stderr parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/d8a79d9c-9506-44f5-9fb4-e1cc35c05e01_characterizeImage_898695_12.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/d8a79d9c-9506-44f5-9fb4-e1cc35c05e01_characterizeImage_898695_12.stderr parsl.dataflow.dflow INFO: Task 517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a423e548ce6d9f6d09379a97f6fb0e1 with result from task 517 parsl.dataflow.dflow DEBUG: Task 517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 518 has memoization hash b5d1fa43497dcb64d8f106d6e1f70be8 parsl.dataflow.memoization INFO: Task 518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 1039 parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/d30afcb6-106a-4009-93c7-a6cd3d5c7733_calibrate_898695_56.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/d30afcb6-106a-4009-93c7-a6cd3d5c7733_calibrate_898695_56.stderr parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/89ec9623-551b-4040-9885-f9211f0fda54_characterizeImage_898695_56.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/89ec9623-551b-4040-9885-f9211f0fda54_characterizeImage_898695_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:09 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af39f386bb93bca2cea42e0de504dbfd with result from task 641 parsl.dataflow.dflow DEBUG: Task 641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/191cc46a-bf97-4846-9105-a64484e049e6_calibrate_898799_62.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/191cc46a-bf97-4846-9105-a64484e049e6_calibrate_898799_62.stderr parsl.dataflow.dflow INFO: Task 863 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93871cbf470e484e1a5a0df7c175a2df with result from task 863 parsl.dataflow.dflow DEBUG: Task 863 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/8e55252a-d2b3-499b-b816-7da745f9001b_calibrate_898696_7.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/8e55252a-d2b3-499b-b816-7da745f9001b_calibrate_898696_7.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aee323f091533fe209bb59d64f7841e5 with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/f0dc286a-344c-4c43-90b4-aebfa5ad6f35_calibrate_898694_44.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/f0dc286a-344c-4c43-90b4-aebfa5ad6f35_calibrate_898694_44.stderr parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6211df7411ed63918753a11b8bc62405 with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/20f9ed90-1f9d-46ba-a4ae-ecb5588576eb_calibrate_898798_54.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/20f9ed90-1f9d-46ba-a4ae-ecb5588576eb_calibrate_898798_54.stderr parsl.dataflow.dflow INFO: Task 916 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f33c95d8dc2eea07f6e0df1da6f1c1cd with result from task 916 parsl.dataflow.dflow DEBUG: Task 916 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 917 has memoization hash e0ce5d3a8dea693e786b6eb550160019 parsl.dataflow.memoization INFO: Task 917 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c290d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 917 try 0 launched on executor multi with executor id 1040 parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/726a9c2c-16a3-47a4-b083-ffb148b41bce_calibrate_898800_14.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/726a9c2c-16a3-47a4-b083-ffb148b41bce_calibrate_898800_14.stderr parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/03ac09a1-68db-4bc4-94f7-acd4f99cc6c5_characterizeImage_898800_14.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/03ac09a1-68db-4bc4-94f7-acd4f99cc6c5_characterizeImage_898800_14.stderr parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 854d45734defbcce16279b91fc63eb99 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/028e8432-65fb-4d28-89ee-faac0ef2df8a_calibrate_898799_8.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/028e8432-65fb-4d28-89ee-faac0ef2df8a_calibrate_898799_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:10 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 935 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6eac639a95c7ad441670a94d81b14a7 with result from task 935 parsl.dataflow.dflow DEBUG: Task 935 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/1417e7f6-7eaa-4d65-9775-90c2c979913b_calibrate_898696_33.stdout parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/1417e7f6-7eaa-4d65-9775-90c2c979913b_calibrate_898696_33.stderr parsl.dataflow.dflow INFO: Task 451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 858b000de5eaec7e698e1d7207d49da8 with result from task 451 parsl.dataflow.dflow DEBUG: Task 451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 452 has memoization hash 750feeb3e623ed64f7c0e47fe20cc744 parsl.dataflow.memoization INFO: Task 452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 1041 parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/cbc0b4f3-0d84-4cc7-a1fa-c863c64cfd76_calibrate_898799_60.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/cbc0b4f3-0d84-4cc7-a1fa-c863c64cfd76_calibrate_898799_60.stderr parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/ab9f05ca-98db-4b35-ab6d-25174f68d32e_characterizeImage_898799_60.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/ab9f05ca-98db-4b35-ab6d-25174f68d32e_characterizeImage_898799_60.stderr parsl.dataflow.dflow INFO: Task 527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87d3eb04140b2039543732483679f5bb with result from task 527 parsl.dataflow.dflow DEBUG: Task 527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/b05e4c1f-55ea-41b5-9752-a0e2d97e5718_calibrate_898694_17.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/b05e4c1f-55ea-41b5-9752-a0e2d97e5718_calibrate_898694_17.stderr parsl.dataflow.dflow INFO: Task 842 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 864ae8a5c32dc3f51f3ac622e2e42ef8 with result from task 842 parsl.dataflow.dflow DEBUG: Task 842 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/957d4535-421d-4fc0-bf2b-ae2507f4bb89_calibrate_898696_14.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/957d4535-421d-4fc0-bf2b-ae2507f4bb89_calibrate_898696_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:10 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4262f38f713e978ae00da2abf3818ee with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/45df528a-d5f9-449a-9f65-789721b06b56_calibrate_898798_41.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/45df528a-d5f9-449a-9f65-789721b06b56_calibrate_898798_41.stderr parsl.dataflow.dflow INFO: Task 610 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9511ae800e79f794c2f5077cf7f69e69 with result from task 610 parsl.dataflow.dflow DEBUG: Task 610 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 611 has memoization hash da6ac486550e063297505f03dd83f23d parsl.dataflow.memoization INFO: Task 611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c510f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 611 try 0 launched on executor multi with executor id 1042 parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/cc55d994-914d-4b84-914c-b0270c3f48fc_calibrate_898696_60.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/cc55d994-914d-4b84-914c-b0270c3f48fc_calibrate_898696_60.stderr parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/78404d00-8116-4989-b1de-a7c334b1dcfe_characterizeImage_898696_60.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/78404d00-8116-4989-b1de-a7c334b1dcfe_characterizeImage_898696_60.stderr parsl.dataflow.dflow INFO: Task 833 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27f92430df6abd6bdc511dd841001be8 with result from task 833 parsl.dataflow.dflow DEBUG: Task 833 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/15537074-b6d6-4793-a6aa-6de8bf8addb4_calibrate_898694_26.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/15537074-b6d6-4793-a6aa-6de8bf8addb4_calibrate_898694_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:11 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ed2c7960d80d130b6c6d53a793dafdf with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/a031f834-08a6-499d-803d-6419e50b7fed_calibrate_898694_29.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/a031f834-08a6-499d-803d-6419e50b7fed_calibrate_898694_29.stderr parsl.dataflow.dflow INFO: Task 1108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd29ca99ae392b91ffef6e4258f8bdf5 with result from task 1108 parsl.dataflow.dflow DEBUG: Task 1108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1109 has memoization hash abd6857da903d1811efc3b37dbcc5ced parsl.dataflow.memoization INFO: Task 1109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c351800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1109 try 0 launched on executor multi with executor id 1043 parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/dda55a02-f2d7-41f8-ab5e-e4a2dde8bcf0_calibrate_898695_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/dda55a02-f2d7-41f8-ab5e-e4a2dde8bcf0_calibrate_898695_53.stderr parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/ca2dbbaf-44e1-4aa6-a926-cee55871bb36_characterizeImage_898695_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/ca2dbbaf-44e1-4aa6-a926-cee55871bb36_characterizeImage_898695_53.stderr parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3948a3fa90e57122ce004d94151db0ac with result from task 343 parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash a3d1d7455bc6f61b1caf63874d1a6e4c parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 1044 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/772d0c16-9e08-4c35-9084-0975b5db18a7_calibrate_898695_54.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/772d0c16-9e08-4c35-9084-0975b5db18a7_calibrate_898695_54.stderr parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/706376c1-bd97-4836-876f-f87627b3fbb5_characterizeImage_898695_54.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/706376c1-bd97-4836-876f-f87627b3fbb5_characterizeImage_898695_54.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af67fbf8eb9abc6a5df2c4a72cb08785 with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/a9b9b203-03db-47de-a126-74b8a1bc2434_calibrate_898695_4.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/a9b9b203-03db-47de-a126-74b8a1bc2434_calibrate_898695_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:11 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b73b651bdda9f703d9377b25e84853f with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash ea6679d188b265f0d66a2b2fd5863c02 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ab600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 1045 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/fd2b2284-dda2-4b03-9bd0-2adea089b3f4_calibrate_898798_46.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/fd2b2284-dda2-4b03-9bd0-2adea089b3f4_calibrate_898798_46.stderr parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ddc1aba8-58a8-4a67-b61c-3002a70819cd_characterizeImage_898798_46.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/ddc1aba8-58a8-4a67-b61c-3002a70819cd_characterizeImage_898798_46.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3c7d70730b0b9763e4d3cee3c5d528d with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/3785c97b-0b7b-4fc5-81e3-83822485c2bc_calibrate_898694_39.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/3785c97b-0b7b-4fc5-81e3-83822485c2bc_calibrate_898694_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:12 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.dataflow.dflow INFO: Task 583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e177f157f271a55972754aa73b11fa9 with result from task 583 parsl.dataflow.dflow DEBUG: Task 583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 584 has memoization hash fc39a5c25bc21ebe633e0d347ff0978e parsl.dataflow.memoization INFO: Task 584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4ded40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 584 try 0 launched on executor multi with executor id 1046 parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/0091b1a4-1eb7-416f-ab47-ff7cb8b00c68_calibrate_898695_21.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/0091b1a4-1eb7-416f-ab47-ff7cb8b00c68_calibrate_898695_21.stderr parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/4328e951-e6db-4b1c-976f-34b0cc418338_characterizeImage_898695_21.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/4328e951-e6db-4b1c-976f-34b0cc418338_characterizeImage_898695_21.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 849755fae16d165a7f1cc1477c3c3cd9 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/d48db75e-100e-4636-a590-5ca468cf221a_calibrate_898799_46.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/d48db75e-100e-4636-a590-5ca468cf221a_calibrate_898799_46.stderr parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c2d45e0d216a520c0db928b962c0a06 with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/b17b7acc-82d4-4d33-a96b-11afc3600de8_calibrate_898800_41.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/b17b7acc-82d4-4d33-a96b-11afc3600de8_calibrate_898800_41.stderr parsl.dataflow.dflow INFO: Task 1064 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c525115006ad3f2af78990eb5e7c5cc7 with result from task 1064 parsl.dataflow.dflow DEBUG: Task 1064 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/b9785b86-4018-4c12-ba5c-4d02f8deca3b_calibrate_898695_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/b9785b86-4018-4c12-ba5c-4d02f8deca3b_calibrate_898695_9.stderr parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f31267956a209f55397bd2aa0c33dcb8 with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/346aefae-4916-4a50-861c-079de9b7154f_calibrate_898798_13.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/346aefae-4916-4a50-861c-079de9b7154f_calibrate_898798_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:13 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.dataflow.dflow INFO: Task 803 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e6b8c4f406f4451b3a6c485285f919d with result from task 803 parsl.dataflow.dflow DEBUG: Task 803 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/1d419d03-2cdb-4d47-83a6-afdb4647f699_calibrate_898694_32.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/1d419d03-2cdb-4d47-83a6-afdb4647f699_calibrate_898694_32.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c68f2fcac0cbdef7d51ef39f5a003d1 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/c4697bb9-d8b6-4cf8-b0e7-0d1e0c95777c_calibrate_898798_24.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/c4697bb9-d8b6-4cf8-b0e7-0d1e0c95777c_calibrate_898798_24.stderr parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4410040e7b00b77e57b0334b160560bd with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/6e27a573-1852-481d-9526-62ec97e73f0b_calibrate_898800_18.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/6e27a573-1852-481d-9526-62ec97e73f0b_calibrate_898800_18.stderr parsl.dataflow.dflow INFO: Task 851 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3ac6c8c5f1c51ae8d0634f21a408df2 with result from task 851 parsl.dataflow.dflow DEBUG: Task 851 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/70b3ab77-65b4-413a-bdfa-4e3c65325dc7_calibrate_898799_56.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/70b3ab77-65b4-413a-bdfa-4e3c65325dc7_calibrate_898799_56.stderr parsl.dataflow.dflow INFO: Task 1112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5700c8fbec033a5df100d2d7f218254d with result from task 1112 parsl.dataflow.dflow DEBUG: Task 1112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/7a13085d-1e1e-44dd-9449-59a0bc7fd0b0_calibrate_898696_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/7a13085d-1e1e-44dd-9449-59a0bc7fd0b0_calibrate_898696_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:14 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a87d311c7482a562ade51501fed9a0e with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/bb9391b3-d020-4abd-889b-33252bd79234_calibrate_898799_32.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/bb9391b3-d020-4abd-889b-33252bd79234_calibrate_898799_32.stderr parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 412e41fc509bb567cef498b6972114cb with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/d2ad87e6-fd09-43dd-9ee9-2385317df624_calibrate_898696_5.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/d2ad87e6-fd09-43dd-9ee9-2385317df624_calibrate_898696_5.stderr parsl.dataflow.dflow INFO: Task 836 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cee341ccb4e9c3a0221b38837164dc80 with result from task 836 parsl.dataflow.dflow DEBUG: Task 836 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/f07dd838-2b9f-4c84-b6a4-07f3351cd576_calibrate_898799_40.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/f07dd838-2b9f-4c84-b6a4-07f3351cd576_calibrate_898799_40.stderr parsl.dataflow.dflow INFO: Task 1016 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e5132214c8bb6b55e5903f4dd5f53bf with result from task 1016 parsl.dataflow.dflow DEBUG: Task 1016 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/efefc570-29c2-424a-82af-c495dfc81f20_calibrate_898696_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/efefc570-29c2-424a-82af-c495dfc81f20_calibrate_898696_45.stderr parsl.dataflow.dflow INFO: Task 673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 980247401cc597902c48e5a6591cdaf6 with result from task 673 parsl.dataflow.dflow DEBUG: Task 673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 674 has memoization hash a720fc71ca2f3943992ac246df8c3148 parsl.dataflow.memoization INFO: Task 674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c374860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 674 try 0 launched on executor multi with executor id 1047 parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/13b6479c-a4fe-4bb5-ae3b-af49bfee5770_calibrate_898694_43.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/13b6479c-a4fe-4bb5-ae3b-af49bfee5770_calibrate_898694_43.stderr parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/8dfdd9dc-5aa7-43b7-8ba0-90a0a6f262ff_characterizeImage_898694_43.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/8dfdd9dc-5aa7-43b7-8ba0-90a0a6f262ff_characterizeImage_898694_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:14 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.dataflow.dflow INFO: Task 634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45d18d5f63d33d03a06c7119cc0b432e with result from task 634 parsl.dataflow.dflow DEBUG: Task 634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 635 has memoization hash 3f018b220b7650b9eaf349698338e420 parsl.dataflow.memoization INFO: Task 635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c512840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 635 try 0 launched on executor multi with executor id 1048 parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/c91421ed-2df9-4855-848b-5d5ed8e3c00e_calibrate_898798_45.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/c91421ed-2df9-4855-848b-5d5ed8e3c00e_calibrate_898798_45.stderr parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/8482838e-3f08-48d6-a578-ee12e41a83cb_characterizeImage_898798_45.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/8482838e-3f08-48d6-a578-ee12e41a83cb_characterizeImage_898798_45.stderr parsl.dataflow.dflow INFO: Task 502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 700562c63a7f19bd1624ed7627dc6aa9 with result from task 502 parsl.dataflow.dflow DEBUG: Task 502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 503 has memoization hash a847d9d9a6bb334b6fa1f06f1c6737cf parsl.dataflow.memoization INFO: Task 503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 1049 parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/345aae2b-7b09-4229-b4ba-5cba7f076cd0_calibrate_898798_58.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/345aae2b-7b09-4229-b4ba-5cba7f076cd0_calibrate_898798_58.stderr parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/75f5eb78-e542-4c2a-8696-47d44f63c5e1_characterizeImage_898798_58.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/75f5eb78-e542-4c2a-8696-47d44f63c5e1_characterizeImage_898798_58.stderr parsl.dataflow.dflow INFO: Task 722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e49a08fb477b3eafc42d7fabb302b499 with result from task 722 parsl.dataflow.dflow DEBUG: Task 722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/b8f5c514-fc37-4654-b10f-fa555465899c_calibrate_898696_6.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/b8f5c514-fc37-4654-b10f-fa555465899c_calibrate_898696_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:15 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8eaaf86139ef20e9f5850c8bce020b0a with result from task 700 parsl.dataflow.dflow DEBUG: Task 700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 701 has memoization hash 2da480e23dd165e01ae48c23d7845006 parsl.dataflow.memoization INFO: Task 701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c375f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 701 try 0 launched on executor multi with executor id 1050 parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/268a55aa-3a60-4331-921e-ab939327c69f_calibrate_898694_19.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/268a55aa-3a60-4331-921e-ab939327c69f_calibrate_898694_19.stderr parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/91f51146-7860-451d-8254-31e894198277_characterizeImage_898694_19.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/91f51146-7860-451d-8254-31e894198277_characterizeImage_898694_19.stderr parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5380e456df56f87db89b9b9b7b6963d with result from task 322 parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 4cfd6e53a2e842a19a45897acd24dfaf parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 1051 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/93e475bd-638b-4b75-a5fa-4d0ca168b5bb_calibrate_898695_19.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/93e475bd-638b-4b75-a5fa-4d0ca168b5bb_calibrate_898695_19.stderr parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/266d3074-4412-48b3-a0d7-3114d0f40f3f_characterizeImage_898695_19.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/266d3074-4412-48b3-a0d7-3114d0f40f3f_characterizeImage_898695_19.stderr parsl.dataflow.dflow INFO: Task 434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7edb1b9fa5b27393e5b2852c0c5dcdf2 with result from task 434 parsl.dataflow.dflow DEBUG: Task 434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/de81f830-7059-48f3-9753-b82c9e9b9ed8_calibrate_898800_53.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/de81f830-7059-48f3-9753-b82c9e9b9ed8_calibrate_898800_53.stderr parsl.dataflow.dflow INFO: Task 1105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 413f031c4d16b405fddbb1a965ba696e with result from task 1105 parsl.dataflow.dflow DEBUG: Task 1105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1106 has memoization hash a788bb505a90dc25fc9b5b190c866ce6 parsl.dataflow.memoization INFO: Task 1106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3514e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1106 try 0 launched on executor multi with executor id 1052 parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/326eb58f-0b1d-496a-8037-d63a1cdfbcff_calibrate_898798_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/326eb58f-0b1d-496a-8037-d63a1cdfbcff_calibrate_898798_43.stderr parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/3b7ee91d-fbb4-4a34-95ad-d8f2f428f744_characterizeImage_898798_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/3b7ee91d-fbb4-4a34-95ad-d8f2f428f744_characterizeImage_898798_43.stderr parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7345f2b2eea36b06aa3d3d6df5fdcb9 with result from task 404 parsl.dataflow.dflow DEBUG: Task 404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/b07c55c9-c8b4-445c-88e8-4824be98fa2f_calibrate_898800_62.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/b07c55c9-c8b4-445c-88e8-4824be98fa2f_calibrate_898800_62.stderr parsl.dataflow.dflow INFO: Task 814 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2787d24ce3a552a407444111619de7fb with result from task 814 parsl.dataflow.dflow DEBUG: Task 814 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 815 has memoization hash 7a91c910ab9f488017aeb13f5563cfbb parsl.dataflow.memoization INFO: Task 815 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 815 try 0 launched on executor multi with executor id 1053 parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/a856c6b5-69ed-4363-8d66-9168615088e3_calibrate_898799_21.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/a856c6b5-69ed-4363-8d66-9168615088e3_calibrate_898799_21.stderr parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/de94a8c9-2e17-4e35-b05d-1858369f5633_characterizeImage_898799_21.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/de94a8c9-2e17-4e35-b05d-1858369f5633_characterizeImage_898799_21.stderr parsl.dataflow.dflow INFO: Task 410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bacbfdf224a4a0d6193d3af9ac502fa with result from task 410 parsl.dataflow.dflow DEBUG: Task 410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/12ca89e5-9b0b-4eb9-ad8c-617dfaddab06_calibrate_898799_7.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/12ca89e5-9b0b-4eb9-ad8c-617dfaddab06_calibrate_898799_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:16 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce7d2bf51d96310945a65b54d94fa4d3 with result from task 655 parsl.dataflow.dflow DEBUG: Task 655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 656 has memoization hash a5b7a3d3394e9b88c975445821d06028 parsl.dataflow.memoization INFO: Task 656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c376d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 656 try 0 launched on executor multi with executor id 1054 parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/327bce8a-87c7-4f8a-878c-1ba825a16f79_calibrate_898800_45.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/327bce8a-87c7-4f8a-878c-1ba825a16f79_calibrate_898800_45.stderr parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/b7f1116a-03a2-4392-b6fa-e343aedfbd10_characterizeImage_898800_45.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/b7f1116a-03a2-4392-b6fa-e343aedfbd10_characterizeImage_898800_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:16 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1993466b4b89a0066597dd7f50acd4bc with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/6fb44d5e-588a-4d92-b63f-f351838f79e5_calibrate_898696_20.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/6fb44d5e-588a-4d92-b63f-f351838f79e5_calibrate_898696_20.stderr parsl.dataflow.dflow INFO: Task 1070 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8023961f245a371fc498fa817bb75a7b with result from task 1070 parsl.dataflow.dflow DEBUG: Task 1070 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/4ed9a6e6-6602-4707-b9b1-b74a36a9a105_calibrate_898696_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/4ed9a6e6-6602-4707-b9b1-b74a36a9a105_calibrate_898696_11.stderr parsl.dataflow.dflow INFO: Task 869 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28a062a8ba39bfee1f6efa496f898a82 with result from task 869 parsl.dataflow.dflow DEBUG: Task 869 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/82e86838-a3e6-4247-87a9-0953a745196b_calibrate_898694_11.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/82e86838-a3e6-4247-87a9-0953a745196b_calibrate_898694_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:17 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18c053846991af12ecc75e8316615e62 with result from task 725 parsl.dataflow.dflow DEBUG: Task 725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/68f983a3-2ec0-46aa-a322-f164bd3bf10d_calibrate_898694_35.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/68f983a3-2ec0-46aa-a322-f164bd3bf10d_calibrate_898694_35.stderr parsl.dataflow.dflow INFO: Task 746 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 096d1ac1a5e400abfa77dfd1d98c9317 with result from task 746 parsl.dataflow.dflow DEBUG: Task 746 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/8cb66809-92a3-4a93-8bd1-52bf5934689d_calibrate_898798_47.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/8cb66809-92a3-4a93-8bd1-52bf5934689d_calibrate_898798_47.stderr parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36c030459edcdb4fc8788be3c87790c5 with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/153f61da-e733-4240-8850-7382fa769764_calibrate_898695_26.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/153f61da-e733-4240-8850-7382fa769764_calibrate_898695_26.stderr parsl.dataflow.dflow INFO: Task 575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c119897be80a5ccec9c3b2e8836c5f34 with result from task 575 parsl.dataflow.dflow DEBUG: Task 575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/130f6928-d5e3-4892-95a5-2e78577a269f_calibrate_898798_20.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/130f6928-d5e3-4892-95a5-2e78577a269f_calibrate_898798_20.stderr parsl.dataflow.dflow INFO: Task 898 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2650775d3468113baf7b2a28af99e0c with result from task 898 parsl.dataflow.dflow DEBUG: Task 898 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 899 has memoization hash 870b62109b471c6aa6253da49b8bf2a3 parsl.dataflow.memoization INFO: Task 899 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c443c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 899 try 0 launched on executor multi with executor id 1055 parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/af4a8e3f-f4a0-46e8-a513-52f595eedb2c_calibrate_898800_56.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/af4a8e3f-f4a0-46e8-a513-52f595eedb2c_calibrate_898800_56.stderr parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/73b24cac-6b8e-44b5-b0e2-2eec52f96872_characterizeImage_898800_56.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/73b24cac-6b8e-44b5-b0e2-2eec52f96872_characterizeImage_898800_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:17 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 941f33860961c311f4faea7e98387c5b with result from task 716 parsl.dataflow.dflow DEBUG: Task 716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/9f133c28-4a07-4999-8fb4-67994fa3640e_calibrate_898695_5.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/9f133c28-4a07-4999-8fb4-67994fa3640e_calibrate_898695_5.stderr parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b60167d21da521478ed68353a61cb5b with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 69e6a497d3b0ee62425293a05ddfaf05 parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d60ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 1056 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/18afa585-c8f2-40c1-82dc-2f662b67e345_calibrate_898800_60.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/18afa585-c8f2-40c1-82dc-2f662b67e345_calibrate_898800_60.stderr parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/6087a1a6-7469-4768-a966-02c72ac53852_characterizeImage_898800_60.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/6087a1a6-7469-4768-a966-02c72ac53852_characterizeImage_898800_60.stderr parsl.dataflow.dflow INFO: Task 988 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 722fcf022c936e3935898a266ad80fd7 with result from task 988 parsl.dataflow.dflow DEBUG: Task 988 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 989 has memoization hash 3dbddd233c41349e770215abb1f436ef parsl.dataflow.memoization INFO: Task 989 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 989 try 0 launched on executor multi with executor id 1057 parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/69253044-9ab6-488f-b610-ee62d01a1524_calibrate_898695_61.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/69253044-9ab6-488f-b610-ee62d01a1524_calibrate_898695_61.stderr parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/1ffb46b2-7a14-495a-9a07-5d8c26b6ad76_characterizeImage_898695_61.stdout parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/1ffb46b2-7a14-495a-9a07-5d8c26b6ad76_characterizeImage_898695_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:18 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 1010 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4ed22444bc18eb2dd2cfa0332bc067e with result from task 1010 parsl.dataflow.dflow DEBUG: Task 1010 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/b454d900-6de6-4f09-9642-5aa3608aee1b_calibrate_898798_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/b454d900-6de6-4f09-9642-5aa3608aee1b_calibrate_898798_8.stderr parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6fba352a342b727deded0e34e29e2fc with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/6699c441-8c53-4e59-ae93-cc5bebfd31b1_calibrate_898799_42.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/6699c441-8c53-4e59-ae93-cc5bebfd31b1_calibrate_898799_42.stderr parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 522af67f60b98c0880b2b6fe504ab6da with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/57b37766-3d7e-4bba-af71-f66c5fa17154_calibrate_898799_5.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/57b37766-3d7e-4bba-af71-f66c5fa17154_calibrate_898799_5.stderr parsl.dataflow.dflow INFO: Task 1046 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0173d496d95ddd7738912cacf3f2a44d with result from task 1046 parsl.dataflow.dflow DEBUG: Task 1046 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/a29b26be-ff63-4599-b333-98b5af42916c_calibrate_898800_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/a29b26be-ff63-4599-b333-98b5af42916c_calibrate_898800_8.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd398edf1a1f39243c5b84cf2ac46151 with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/77bb9edd-d18b-43b8-bc40-cebb018f936c_calibrate_898798_33.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/77bb9edd-d18b-43b8-bc40-cebb018f936c_calibrate_898798_33.stderr parsl.dataflow.dflow INFO: Task 1055 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 087d210ab0934db19a6f60fa57ada051 with result from task 1055 parsl.dataflow.dflow DEBUG: Task 1055 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/61d049ea-ee5f-4342-bbe9-884ca8b5b4cd_calibrate_898800_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/61d049ea-ee5f-4342-bbe9-884ca8b5b4cd_calibrate_898800_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:19 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 778826edf612e36b143b81837c8995bd with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/18220d26-02dc-46a5-92ac-efb380e63ef8_calibrate_898799_18.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/18220d26-02dc-46a5-92ac-efb380e63ef8_calibrate_898799_18.stderr parsl.dataflow.dflow INFO: Task 977 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3b8606a258fdcb0abd19d8f1a6187d6 with result from task 977 parsl.dataflow.dflow DEBUG: Task 977 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/e3370a5b-cd5a-42f2-b070-7dfb3c3c6d70_calibrate_898799_57.stdout parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/e3370a5b-cd5a-42f2-b070-7dfb3c3c6d70_calibrate_898799_57.stderr parsl.dataflow.dflow INFO: Task 1013 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33979d3e6fec80f791593101d5a0b5ba with result from task 1013 parsl.dataflow.dflow DEBUG: Task 1013 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/c7896c29-b892-473b-b967-04e48acab913_calibrate_898696_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/c7896c29-b892-473b-b967-04e48acab913_calibrate_898696_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:19 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90057db75adb2adb25cc4560cd4cf09a with result from task 376 parsl.dataflow.dflow DEBUG: Task 376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 377 has memoization hash a9fb7222e1834314661c00a410cc99b2 parsl.dataflow.memoization INFO: Task 377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6616c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 1058 parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/add8b1de-a1e0-4886-9af8-aa236e1e9f0d_calibrate_898694_40.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/add8b1de-a1e0-4886-9af8-aa236e1e9f0d_calibrate_898694_40.stderr parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/485c3048-52b1-49fb-b370-d22677c14191_characterizeImage_898694_40.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/485c3048-52b1-49fb-b370-d22677c14191_characterizeImage_898694_40.stderr parsl.dataflow.dflow INFO: Task 695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 363a86b20c6f2c588d3eecaee0e6c98e with result from task 695 parsl.dataflow.dflow DEBUG: Task 695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/7a09d777-e987-44c8-bf2e-2c6cab429a1e_calibrate_898694_23.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/7a09d777-e987-44c8-bf2e-2c6cab429a1e_calibrate_898694_23.stderr parsl.dataflow.dflow INFO: Task 562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cc950de48dba5e9683430b2f3206cf0 with result from task 562 parsl.dataflow.dflow DEBUG: Task 562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 563 has memoization hash 7d05c08ea51a30ad4eb2185abd081efd parsl.dataflow.memoization INFO: Task 563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4ddb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 563 try 0 launched on executor multi with executor id 1059 parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/a5929016-1a94-46b4-a7b2-79d74e537e2b_calibrate_898694_21.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/a5929016-1a94-46b4-a7b2-79d74e537e2b_calibrate_898694_21.stderr parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/7a3c6910-57fc-4a81-bbb3-85a8417d66fa_characterizeImage_898694_21.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/7a3c6910-57fc-4a81-bbb3-85a8417d66fa_characterizeImage_898694_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:20 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 1027 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfd041640aec2f298875f0b63b424e0d with result from task 1027 parsl.dataflow.dflow DEBUG: Task 1027 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1028 has memoization hash 2461f9ea5f7bf6f066ce2e7c9d093fa4 parsl.dataflow.memoization INFO: Task 1028 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c328360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1028 try 0 launched on executor multi with executor id 1060 parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/d81ff872-acd5-4a20-b0d6-3f1de3120f30_calibrate_898694_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/d81ff872-acd5-4a20-b0d6-3f1de3120f30_calibrate_898694_13.stderr parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/13fcdb83-0b8c-4bb4-aa5d-f9134b1eb91f_characterizeImage_898694_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/13fcdb83-0b8c-4bb4-aa5d-f9134b1eb91f_characterizeImage_898694_13.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7a28519283f41eacdbfa55a69ebf6fe with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash c74183afd69a4f46d7614b52ca098f6a parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5aa660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 1061 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/fba4e545-b55a-4abd-b6da-a231cf963a22_calibrate_898800_58.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/fba4e545-b55a-4abd-b6da-a231cf963a22_calibrate_898800_58.stderr parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/d86e7923-f138-475d-a237-ecd482f3996f_characterizeImage_898800_58.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/d86e7923-f138-475d-a237-ecd482f3996f_characterizeImage_898800_58.stderr parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dfe100dacb8d85249fb523a9f1cbd51 with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/dffc3535-62ab-4ef3-af36-ff45996559f3_calibrate_898695_29.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/dffc3535-62ab-4ef3-af36-ff45996559f3_calibrate_898695_29.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 735ceba1660b520539875da40fc9dd22 with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/3eecc118-d7b5-4a0c-be0e-9d752d60ed64_calibrate_898695_60.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/3eecc118-d7b5-4a0c-be0e-9d752d60ed64_calibrate_898695_60.stderr parsl.dataflow.dflow INFO: Task 893 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a5518d69f95a24a0fcc8a0af8e615b7 with result from task 893 parsl.dataflow.dflow DEBUG: Task 893 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/d9565533-4f3f-4fc8-853d-13085f870ac3_calibrate_898798_59.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/d9565533-4f3f-4fc8-853d-13085f870ac3_calibrate_898798_59.stderr parsl.dataflow.dflow INFO: Task 373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdcdc25d93577d945ab35ba388319ed4 with result from task 373 parsl.dataflow.dflow DEBUG: Task 373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 374 has memoization hash e45f9f10518590763731cad502ec5d78 parsl.dataflow.memoization INFO: Task 374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d6613a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 374 try 0 launched on executor multi with executor id 1062 parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/8532ddc3-7d2a-410e-98d1-cff0f3cc0f7c_calibrate_898696_43.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/8532ddc3-7d2a-410e-98d1-cff0f3cc0f7c_calibrate_898696_43.stderr parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/a1772282-ee4f-4e83-83af-cebb66455c00_characterizeImage_898696_43.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/a1772282-ee4f-4e83-83af-cebb66455c00_characterizeImage_898696_43.stderr parsl.dataflow.dflow INFO: Task 754 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4a7fb5aea04dfbdbed2ead0171c0871 with result from task 754 parsl.dataflow.dflow DEBUG: Task 754 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 755 has memoization hash 71c1df5265c6a7e5d73bca960a81b496 parsl.dataflow.memoization INFO: Task 755 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3ba480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 755 try 0 launched on executor multi with executor id 1063 parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/e139a0dc-d854-4395-9af3-1a5852ba4a41_calibrate_898694_20.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/e139a0dc-d854-4395-9af3-1a5852ba4a41_calibrate_898694_20.stderr parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/7110b934-b1d4-4033-bd8c-020a49556d6d_characterizeImage_898694_20.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/7110b934-b1d4-4033-bd8c-020a49556d6d_characterizeImage_898694_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:21 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 775 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27df080f51b282c70d56bd29fab5b7a1 with result from task 775 parsl.dataflow.dflow DEBUG: Task 775 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 776 has memoization hash e782f0819ec666a0a24cfdb39a2a9b10 parsl.dataflow.memoization INFO: Task 776 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 776 try 0 launched on executor multi with executor id 1064 parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/6fa1469d-f3c8-4445-bbf8-d9611363b150_calibrate_898694_4.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/6fa1469d-f3c8-4445-bbf8-d9611363b150_calibrate_898694_4.stderr parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/46d8fcbc-bd6f-429c-8be5-f8940033e295_characterizeImage_898694_4.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/46d8fcbc-bd6f-429c-8be5-f8940033e295_characterizeImage_898694_4.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 291aa7f4a8b6e8db89cedde669f07141 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/f05034f5-bf2c-4425-9c65-2a5244c29081_calibrate_898798_6.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/f05034f5-bf2c-4425-9c65-2a5244c29081_calibrate_898798_6.stderr parsl.dataflow.dflow INFO: Task 1067 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b37a3dc89a7eb1c0d6ad59523dcbec7 with result from task 1067 parsl.dataflow.dflow DEBUG: Task 1067 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/9e8d8f81-dab4-4261-a1d3-8355cf83d399_calibrate_898798_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/9e8d8f81-dab4-4261-a1d3-8355cf83d399_calibrate_898798_37.stderr parsl.dataflow.dflow INFO: Task 1022 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4841ce4d6d45d8be344c9ec419dd7f0 with result from task 1022 parsl.dataflow.dflow DEBUG: Task 1022 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/b136bfd4-8230-4a28-a139-1404d176f03e_calibrate_898695_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/b136bfd4-8230-4a28-a139-1404d176f03e_calibrate_898695_33.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 305 try 1 failed parsl.dataflow.dflow ERROR: Task 305 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 813d931c046c9836b74a99f4f389670b with result from task 305 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/377242ec-5005-4e58-82c4-26bd207a9818_calibrate_898694_14.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/377242ec-5005-4e58-82c4-26bd207a9818_calibrate_898694_14.stderr parsl.dataflow.dflow INFO: Task 512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c9784ca3982c966bdf1306df3e199f7 with result from task 512 parsl.dataflow.dflow DEBUG: Task 512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/04b032ee-0b81-4b2a-bea8-5c692d74a988_calibrate_898799_11.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/04b032ee-0b81-4b2a-bea8-5c692d74a988_calibrate_898799_11.stderr parsl.dataflow.dflow INFO: Task 941 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2cc7df5b82607c08731b0ca8a98c675 with result from task 941 parsl.dataflow.dflow DEBUG: Task 941 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/7f8e6443-4c15-42b3-8ddd-ba570570a884_calibrate_898798_21.stdout parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/7f8e6443-4c15-42b3-8ddd-ba570570a884_calibrate_898798_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:21 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50167c6be7de79bb172c7f303d03abc5 with result from task 691 parsl.dataflow.dflow DEBUG: Task 691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 692 has memoization hash aa5b57bbbdccc25e47882d98dfe89330 parsl.dataflow.memoization INFO: Task 692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c375a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 692 try 0 launched on executor multi with executor id 1065 parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/66882944-caeb-4f41-ac36-7157c5718603_calibrate_898695_34.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/66882944-caeb-4f41-ac36-7157c5718603_calibrate_898695_34.stderr parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/9473746a-381a-42a6-a0a6-d63594874c2b_characterizeImage_898695_34.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/9473746a-381a-42a6-a0a6-d63594874c2b_characterizeImage_898695_34.stderr parsl.dataflow.dflow INFO: Task 965 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2abef293a75b13a457be0049b4eb5977 with result from task 965 parsl.dataflow.dflow DEBUG: Task 965 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/b728b248-858e-4358-be28-1829ce4ca35b_calibrate_898799_51.stdout parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/b728b248-858e-4358-be28-1829ce4ca35b_calibrate_898799_51.stderr parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6530e4c8508cbaf9358727c2eac7fcc8 with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/c89110c6-4ebc-4113-8f35-3e4027170324_calibrate_898798_23.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/c89110c6-4ebc-4113-8f35-3e4027170324_calibrate_898798_23.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5002cbf74361ad86f88aae058c5ba5f0 with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/b35ce77d-fa8a-450b-860d-daf9f70a9ecd_calibrate_898694_55.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/b35ce77d-fa8a-450b-860d-daf9f70a9ecd_calibrate_898694_55.stderr parsl.dataflow.dflow INFO: Task 986 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7c0232eedc1803bf91eae0d4a049fa6 with result from task 986 parsl.dataflow.dflow DEBUG: Task 986 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/a37af44c-7028-450a-99f4-d2d587bef4b7_calibrate_898798_12.stdout parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/a37af44c-7028-450a-99f4-d2d587bef4b7_calibrate_898798_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:22 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68f488310c05882531bfd00b896c5f15 with result from task 506 parsl.dataflow.dflow DEBUG: Task 506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/33486b45-81b7-4c99-876a-1ce51f88cd31_calibrate_898800_37.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/33486b45-81b7-4c99-876a-1ce51f88cd31_calibrate_898800_37.stderr parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f942578253041c274046cd4b61af18da with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/6d8b84ad-d86a-4417-870e-1041bba36268_calibrate_898799_4.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/6d8b84ad-d86a-4417-870e-1041bba36268_calibrate_898799_4.stderr parsl.dataflow.dflow INFO: Task 1075 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e342e6697b420d831749ba5fd274c4a9 with result from task 1075 parsl.dataflow.dflow DEBUG: Task 1075 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1076 has memoization hash f2aa6b814cae5b6cff944ebcddf7bc13 parsl.dataflow.memoization INFO: Task 1076 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3298a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1076 try 0 launched on executor multi with executor id 1066 parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/315e6269-759d-4f3e-950c-b000f419b0c5_calibrate_898695_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/315e6269-759d-4f3e-950c-b000f419b0c5_calibrate_898695_15.stderr parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/1795edc5-ea45-441b-9448-94efa20a8a89_characterizeImage_898695_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/1795edc5-ea45-441b-9448-94efa20a8a89_characterizeImage_898695_15.stderr parsl.dataflow.dflow INFO: Task 589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 727dc94e0a4e098a243f8648c1a47278 with result from task 589 parsl.dataflow.dflow DEBUG: Task 589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 590 has memoization hash 1ac4f5c2bf063a5efa2692a0b0853b14 parsl.dataflow.memoization INFO: Task 590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4df380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 590 try 0 launched on executor multi with executor id 1067 parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/32aa74c9-e928-4db8-9593-f9165e0185f5_calibrate_898695_24.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/32aa74c9-e928-4db8-9593-f9165e0185f5_calibrate_898695_24.stderr parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/679acd56-4331-47fd-a2a1-2b9bb69f70a3_characterizeImage_898695_24.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/679acd56-4331-47fd-a2a1-2b9bb69f70a3_characterizeImage_898695_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:23 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dce3c26bb613b8350e34998c2b76c9f4 with result from task 520 parsl.dataflow.dflow DEBUG: Task 520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 521 has memoization hash 11ddcafe2f5f9ee40faa626adb7e37f6 parsl.dataflow.memoization INFO: Task 521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 521 try 0 launched on executor multi with executor id 1068 parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/6c9a0081-fd39-441b-b2f1-f4d4b497e8b4_calibrate_898799_44.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/6c9a0081-fd39-441b-b2f1-f4d4b497e8b4_calibrate_898799_44.stderr parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/6e8b5693-e5f1-48cb-af12-051a79254362_characterizeImage_898799_44.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/6e8b5693-e5f1-48cb-af12-051a79254362_characterizeImage_898799_44.stderr parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08168256ac451cc5efdb5f4486694f51 with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/2a08d6f7-5920-4ef8-b35c-8485fbbcf827_calibrate_898799_27.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/2a08d6f7-5920-4ef8-b35c-8485fbbcf827_calibrate_898799_27.stderr parsl.dataflow.dflow INFO: Task 578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7556aa1912fa5b2f431d3b0e3b0f78c6 with result from task 578 parsl.dataflow.dflow DEBUG: Task 578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/cf1cba2e-43ca-45ba-965b-88d4c1a0f07c_calibrate_898695_16.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/cf1cba2e-43ca-45ba-965b-88d4c1a0f07c_calibrate_898695_16.stderr parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eea46792047603e13f83860f452f59ee with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/794222b2-e11b-4dd8-a760-c746fac8f672_calibrate_898696_55.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/794222b2-e11b-4dd8-a760-c746fac8f672_calibrate_898696_55.stderr parsl.dataflow.dflow INFO: Task 529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb8802b478b93bdcd650e1d548296994 with result from task 529 parsl.dataflow.dflow DEBUG: Task 529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 530 has memoization hash 415a9d92fc459545afdf36c7a6134347 parsl.dataflow.memoization INFO: Task 530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 530 try 0 launched on executor multi with executor id 1069 parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/aa30f444-6777-4311-9acb-263508b44d0a_calibrate_898696_23.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/aa30f444-6777-4311-9acb-263508b44d0a_calibrate_898696_23.stderr parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/c3b350e3-39a2-4c33-ae6a-05333da96e9e_characterizeImage_898696_23.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/c3b350e3-39a2-4c33-ae6a-05333da96e9e_characterizeImage_898696_23.stderr parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d09b21d6ab07e0164bc62f6ffbf3a9a6 with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash 544fb403a8fa0754c5da5c30fde66127 parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 1070 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/ec251919-71d9-4b1d-8ead-98757f07921f_calibrate_898799_45.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/ec251919-71d9-4b1d-8ead-98757f07921f_calibrate_898799_45.stderr parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/592c88e9-5f66-4730-9495-af7ba3b0bc9e_characterizeImage_898799_45.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/592c88e9-5f66-4730-9495-af7ba3b0bc9e_characterizeImage_898799_45.stderr parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b74f48c3411c403d315fbbccf354565d with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash 50c0c9f57577354a9fef86c07aaa78ba parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d660d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 1071 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/5d9fb8c4-d209-4762-a403-ac39853f47d4_calibrate_898695_40.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/5d9fb8c4-d209-4762-a403-ac39853f47d4_calibrate_898695_40.stderr parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/5bf44745-612d-4575-bc47-64996e12fe68_characterizeImage_898695_40.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/5bf44745-612d-4575-bc47-64996e12fe68_characterizeImage_898695_40.stderr parsl.dataflow.dflow INFO: Task 890 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6413df4da395c56bd59a9c16f16b8c46 with result from task 890 parsl.dataflow.dflow DEBUG: Task 890 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/07841f10-d3d6-410c-a4f9-af249370600f_calibrate_898694_52.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/07841f10-d3d6-410c-a4f9-af249370600f_calibrate_898694_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:23 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed9d1408999177723e0f5283c03acc42 with result from task 491 parsl.dataflow.dflow DEBUG: Task 491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/75f29b4a-3a0e-4785-b51f-87f276a46cce_calibrate_898798_62.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/75f29b4a-3a0e-4785-b51f-87f276a46cce_calibrate_898798_62.stderr parsl.dataflow.dflow INFO: Task 938 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4db653bb5ba760d363d3e2b71f284697 with result from task 938 parsl.dataflow.dflow DEBUG: Task 938 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/f7b5104d-046c-47b0-a00f-ba1593daf00b_calibrate_898695_28.stdout parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/f7b5104d-046c-47b0-a00f-ba1593daf00b_calibrate_898695_28.stderr parsl.dataflow.dflow INFO: Task 959 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a69308867f2396776a442fc4746025f with result from task 959 parsl.dataflow.dflow DEBUG: Task 959 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/2d5aa007-4806-4261-8a14-8e2434ddde20_calibrate_898798_51.stdout parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/2d5aa007-4806-4261-8a14-8e2434ddde20_calibrate_898798_51.stderr parsl.dataflow.dflow INFO: Task 629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ca74f45a2e1bac4bce88c0083ac3395 with result from task 629 parsl.dataflow.dflow DEBUG: Task 629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/4d5a717f-48fa-4bef-adfd-ca7094cef6b4_calibrate_898799_34.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/4d5a717f-48fa-4bef-adfd-ca7094cef6b4_calibrate_898799_34.stderr parsl.dataflow.dflow INFO: Task 770 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6632e7a1fa5c99af4945d3f09f0964b8 with result from task 770 parsl.dataflow.dflow DEBUG: Task 770 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/4ab11c6a-a28f-4e2d-ac12-cb21c0241fd9_calibrate_898800_10.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/4ab11c6a-a28f-4e2d-ac12-cb21c0241fd9_calibrate_898800_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:24 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 995 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0007b0a4bd4c3234e043bec1a59bbd8 with result from task 995 parsl.dataflow.dflow DEBUG: Task 995 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/fe723d74-e386-4b53-99d5-35d50ac2cc83_calibrate_898798_7.stdout parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/fe723d74-e386-4b53-99d5-35d50ac2cc83_calibrate_898798_7.stderr parsl.dataflow.dflow INFO: Task 886 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9c9530ce9cc4c1479faf80adc44cb01 with result from task 886 parsl.dataflow.dflow DEBUG: Task 886 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 887 has memoization hash 24d8008216a733d30f5754c2aa671137 parsl.dataflow.memoization INFO: Task 887 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c442fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 887 try 0 launched on executor multi with executor id 1072 parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/d88de9c2-212c-46e0-817e-b88fe3722b6e_calibrate_898799_14.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/d88de9c2-212c-46e0-817e-b88fe3722b6e_calibrate_898799_14.stderr parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/63ce5329-c736-467a-92c4-e84a9730e459_characterizeImage_898799_14.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/63ce5329-c736-467a-92c4-e84a9730e459_characterizeImage_898799_14.stderr parsl.dataflow.dflow INFO: Task 919 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82812a34f76104b2922b44e87a037a5e with result from task 919 parsl.dataflow.dflow DEBUG: Task 919 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 920 has memoization hash f7739606b6d9e735452b63c5495b39d1 parsl.dataflow.memoization INFO: Task 920 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c291080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 920 try 0 launched on executor multi with executor id 1073 parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/72117df8-53ff-49ac-bfb5-7725f8896c2d_calibrate_898800_5.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/72117df8-53ff-49ac-bfb5-7725f8896c2d_calibrate_898800_5.stderr parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/edf29467-f313-40b0-b03d-667ff3933dfd_characterizeImage_898800_5.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/edf29467-f313-40b0-b03d-667ff3933dfd_characterizeImage_898800_5.stderr parsl.dataflow.dflow INFO: Task 413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c3beb8f33debb714d4fd6b62366ddc0 with result from task 413 parsl.dataflow.dflow DEBUG: Task 413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/177bd906-a474-4156-bbdf-4b07978939db_calibrate_898694_6.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/177bd906-a474-4156-bbdf-4b07978939db_calibrate_898694_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:25 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7d2dcd29cdd305c4d9aaa60adec061e with result from task 647 parsl.dataflow.dflow DEBUG: Task 647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/3ca52e6a-1bab-47dd-9992-f13c688a233f_calibrate_898799_3.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/3ca52e6a-1bab-47dd-9992-f13c688a233f_calibrate_898799_3.stderr parsl.dataflow.dflow INFO: Task 1115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b024ce4023e23df1cdcb653bc610a5a with result from task 1115 parsl.dataflow.dflow DEBUG: Task 1115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/440fc355-368e-46c1-9115-f7eb8f958d50_calibrate_898799_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/440fc355-368e-46c1-9115-f7eb8f958d50_calibrate_898799_20.stderr parsl.dataflow.dflow INFO: Task 820 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a49633e619abcde7421b88f067b3b49 with result from task 820 parsl.dataflow.dflow DEBUG: Task 820 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 821 has memoization hash 39a8d98c4e54ff2b9cafee43c51a4baf parsl.dataflow.memoization INFO: Task 821 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 821 try 0 launched on executor multi with executor id 1074 parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/cadb0179-1aed-4e65-b6d8-4b309602fe2b_calibrate_898696_51.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/cadb0179-1aed-4e65-b6d8-4b309602fe2b_calibrate_898696_51.stderr parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/2f0434f7-8a50-4bfb-922d-11e4088ed3a5_characterizeImage_898696_51.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/2f0434f7-8a50-4bfb-922d-11e4088ed3a5_characterizeImage_898696_51.stderr parsl.dataflow.dflow INFO: Task 731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50fa1493679dc160276cfd800eb10747 with result from task 731 parsl.dataflow.dflow DEBUG: Task 731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/375b1dcd-c1ea-439b-84ee-2a97b8c3d192_calibrate_898696_50.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/375b1dcd-c1ea-439b-84ee-2a97b8c3d192_calibrate_898696_50.stderr parsl.dataflow.dflow INFO: Task 829 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac8acbe0ced07e07d16c0bc44f42eada with result from task 829 parsl.dataflow.dflow DEBUG: Task 829 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 830 has memoization hash ca8ec6159a3844b891fbb6bde691dea7 parsl.dataflow.memoization INFO: Task 830 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 830 try 0 launched on executor multi with executor id 1075 parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/7c7bb6bc-14bd-47b3-b254-4fe50d508dd2_calibrate_898694_34.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/7c7bb6bc-14bd-47b3-b254-4fe50d508dd2_calibrate_898694_34.stderr parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/bb7619d4-3a63-4078-82f9-3bfe7d260ebc_characterizeImage_898694_34.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/bb7619d4-3a63-4078-82f9-3bfe7d260ebc_characterizeImage_898694_34.stderr parsl.dataflow.dflow INFO: Task 1100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5315802b5ac4163b8af7327d07716eaa with result from task 1100 parsl.dataflow.dflow DEBUG: Task 1100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/8be85f31-09d4-4e82-b21d-027897835edb_calibrate_898800_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/8be85f31-09d4-4e82-b21d-027897835edb_calibrate_898800_50.stderr parsl.dataflow.dflow INFO: Task 818 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a3de713a6b08f729e361760de5f5a1e with result from task 818 parsl.dataflow.dflow DEBUG: Task 818 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/1f0b242d-03d9-4e1f-846c-677e7cbbca31_calibrate_898799_35.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/1f0b242d-03d9-4e1f-846c-677e7cbbca31_calibrate_898799_35.stderr parsl.dataflow.dflow INFO: Task 380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f32ed1802853a727f49734310071b183 with result from task 380 parsl.dataflow.dflow DEBUG: Task 380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/3e9d780e-bcb7-4a31-b311-3da39b8b2275_calibrate_898798_38.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/3e9d780e-bcb7-4a31-b311-3da39b8b2275_calibrate_898798_38.stderr parsl.dataflow.dflow INFO: Task 1103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c399e3d51252a3e861efb77e9974e07 with result from task 1103 parsl.dataflow.dflow DEBUG: Task 1103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/80dfd7a1-620e-47f1-b12b-c56d0f78e28b_calibrate_898695_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/80dfd7a1-620e-47f1-b12b-c56d0f78e28b_calibrate_898695_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:25 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86cad469e35658f3cc859021e08ed78f with result from task 572 parsl.dataflow.dflow DEBUG: Task 572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/11c2656e-7c3a-4b74-bb0f-cb52ac438f1c_calibrate_898799_26.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/11c2656e-7c3a-4b74-bb0f-cb52ac438f1c_calibrate_898799_26.stderr parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a89b993a9de4fbc2bc06587f5519482 with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/e3aed31e-40bd-4c61-9a5d-64856ecbf96a_calibrate_898695_41.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/e3aed31e-40bd-4c61-9a5d-64856ecbf96a_calibrate_898695_41.stderr parsl.dataflow.dflow INFO: Task 719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50bc805d3a349b6ce51cfde3a4fe26de with result from task 719 parsl.dataflow.dflow DEBUG: Task 719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/f2087b2c-2eca-43b6-b779-4e1af7a975e4_calibrate_898798_39.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/f2087b2c-2eca-43b6-b779-4e1af7a975e4_calibrate_898798_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:26 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d7d9c95c5c00ef95f2123a8f4702f53 with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/78f801df-8bad-474a-83c9-3519913117e2_calibrate_898798_4.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/78f801df-8bad-474a-83c9-3519913117e2_calibrate_898798_4.stderr parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aef49d492db63ed87c2761e2d5fdcc03 with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash 0c86735254fbf9f1bb8229c31ba15684 parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d660720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 1076 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/dc438aad-ed44-4aa3-a905-514717db4f0a_calibrate_898695_27.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/dc438aad-ed44-4aa3-a905-514717db4f0a_calibrate_898695_27.stderr parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/c3073a09-79de-4a71-9f5e-a203f0a3768a_characterizeImage_898695_27.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/c3073a09-79de-4a71-9f5e-a203f0a3768a_characterizeImage_898695_27.stderr parsl.dataflow.dflow INFO: Task 557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5162e534511119e18d9eb52cf0895183 with result from task 557 parsl.dataflow.dflow DEBUG: Task 557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/58e14350-bc40-4e2f-87a3-550c6939bfae_calibrate_898800_42.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/58e14350-bc40-4e2f-87a3-550c6939bfae_calibrate_898800_42.stderr parsl.dataflow.dflow INFO: Task 509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af458c1b888f8c4e3992266b509339a8 with result from task 509 parsl.dataflow.dflow DEBUG: Task 509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/75da0acb-6ca5-46bf-afa6-12c20917cc52_calibrate_898799_39.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/75da0acb-6ca5-46bf-afa6-12c20917cc52_calibrate_898799_39.stderr parsl.dataflow.dflow INFO: Task 620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc6d6c044adfe25e5f76f960bd705932 with result from task 620 parsl.dataflow.dflow DEBUG: Task 620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/2e42dd95-8021-463d-98f9-761ef163ba9f_calibrate_898800_38.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/2e42dd95-8021-463d-98f9-761ef163ba9f_calibrate_898800_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:27 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9eae98ca9224320399426d9d708d5d4 with result from task 650 parsl.dataflow.dflow DEBUG: Task 650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/102c168a-1de2-467f-a08f-30c9b105f6f7_calibrate_898798_18.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/102c168a-1de2-467f-a08f-30c9b105f6f7_calibrate_898798_18.stderr parsl.dataflow.dflow INFO: Task 386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 783c126fd67ea9f9317890f56a1d8097 with result from task 386 parsl.dataflow.dflow DEBUG: Task 386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/e66cc113-8142-46f7-ab2f-f3e4fea42a03_calibrate_898800_26.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/e66cc113-8142-46f7-ab2f-f3e4fea42a03_calibrate_898800_26.stderr parsl.dataflow.dflow INFO: Task 443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 825e57f9d13260095333cc6d58553367 with result from task 443 parsl.dataflow.dflow DEBUG: Task 443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/d98b3909-7102-45b6-845a-c340ad39ff03_calibrate_898798_50.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/d98b3909-7102-45b6-845a-c340ad39ff03_calibrate_898798_50.stderr parsl.dataflow.dflow INFO: Task 1004 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a1b6f50640eae038defe18fd6fa5cf0 with result from task 1004 parsl.dataflow.dflow DEBUG: Task 1004 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/80f481bd-aeb1-4c15-aad4-b47936b66dea_calibrate_898799_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/80f481bd-aeb1-4c15-aad4-b47936b66dea_calibrate_898799_53.stderr parsl.dataflow.dflow INFO: Task 760 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb1536b7ee76f2b48e6a1c46d480cee2 with result from task 760 parsl.dataflow.dflow DEBUG: Task 760 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 761 has memoization hash 158011859d6499de22593cd90fd217a5 parsl.dataflow.memoization INFO: Task 761 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3baac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 761 try 0 launched on executor multi with executor id 1077 parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/a4b1a521-e556-42cd-bffa-4b0454972074_calibrate_898799_38.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/a4b1a521-e556-42cd-bffa-4b0454972074_calibrate_898799_38.stderr parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/981007ea-b1bc-4d0e-90ec-94715bb9e83c_characterizeImage_898799_38.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/981007ea-b1bc-4d0e-90ec-94715bb9e83c_characterizeImage_898799_38.stderr parsl.dataflow.dflow INFO: Task 676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b00d60a8b2d4de58447168ea5c322b7 with result from task 676 parsl.dataflow.dflow DEBUG: Task 676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 677 has memoization hash e3e3edd96503b13afda0b5a95429845d parsl.dataflow.memoization INFO: Task 677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c374b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 677 try 0 launched on executor multi with executor id 1078 parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/d3088256-e68b-4fa6-ba5a-91bd5771a5e9_calibrate_898695_1.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/d3088256-e68b-4fa6-ba5a-91bd5771a5e9_calibrate_898695_1.stderr parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/3b9cf0b0-6304-4fcd-a085-ca8f809d8957_characterizeImage_898695_1.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/3b9cf0b0-6304-4fcd-a085-ca8f809d8957_characterizeImage_898695_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:27 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 901 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e2097327dfec86645c84278b85823be with result from task 901 parsl.dataflow.dflow DEBUG: Task 901 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 902 has memoization hash 9547a0d715b8dee055d6c9afe8bbbb2f parsl.dataflow.memoization INFO: Task 902 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c443d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 902 try 0 launched on executor multi with executor id 1079 parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/c01c12ae-af7c-4fcd-8284-37c2b4804bf5_calibrate_898800_34.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/c01c12ae-af7c-4fcd-8284-37c2b4804bf5_calibrate_898800_34.stderr parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/5d91dbe4-b071-4fee-aeaa-671ec9123def_characterizeImage_898800_34.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/5d91dbe4-b071-4fee-aeaa-671ec9123def_characterizeImage_898800_34.stderr parsl.dataflow.dflow INFO: Task 740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 924075db6b2ef22f95a4bc3711303167 with result from task 740 parsl.dataflow.dflow DEBUG: Task 740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/07c62024-7d8e-40ac-951f-dd5bb06ea7f3_calibrate_898694_49.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/07c62024-7d8e-40ac-951f-dd5bb06ea7f3_calibrate_898694_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:28 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9ee7c9f2e7a230b2472350db4c9c9af with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/2290919f-e4bf-4a7b-8fc2-646bbbc26258_calibrate_898694_25.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/2290919f-e4bf-4a7b-8fc2-646bbbc26258_calibrate_898694_25.stderr parsl.dataflow.dflow INFO: Task 626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73d1f2071bad4ecd319701d32ac4273c with result from task 626 parsl.dataflow.dflow DEBUG: Task 626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/00208e56-f082-4665-ba5b-d43c89b8d4c6_calibrate_898694_28.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/00208e56-f082-4665-ba5b-d43c89b8d4c6_calibrate_898694_28.stderr parsl.dataflow.dflow INFO: Task 1052 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e41336a2472a7f8a0cc5140c3a75604 with result from task 1052 parsl.dataflow.dflow DEBUG: Task 1052 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/e42dbf8d-bbe0-4398-af78-96ee9fb90d83_calibrate_898800_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/e42dbf8d-bbe0-4398-af78-96ee9fb90d83_calibrate_898800_9.stderr parsl.dataflow.dflow INFO: Task 623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e20c24f5a0533c6039c735bb0f18123 with result from task 623 parsl.dataflow.dflow DEBUG: Task 623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/fa9f52c3-c230-46d4-a2fc-be4c4f85602f_calibrate_898696_4.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/fa9f52c3-c230-46d4-a2fc-be4c4f85602f_calibrate_898696_4.stderr parsl.dataflow.dflow INFO: Task 929 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9ec3bc735c12ca0752854d7825c3344 with result from task 929 parsl.dataflow.dflow DEBUG: Task 929 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/517ede20-ed68-4558-ae68-0d27e70df0f6_calibrate_898696_16.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/517ede20-ed68-4558-ae68-0d27e70df0f6_calibrate_898696_16.stderr parsl.dataflow.dflow INFO: Task 644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2edda1b4886e1e77af95bc8532d963b with result from task 644 parsl.dataflow.dflow DEBUG: Task 644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/4faadd51-c882-4f86-9de9-fcf3aaa0ed8a_calibrate_898798_11.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/4faadd51-c882-4f86-9de9-fcf3aaa0ed8a_calibrate_898798_11.stderr parsl.dataflow.dflow INFO: Task 968 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cb38879c440d7edb10abe1d4bfc57b6 with result from task 968 parsl.dataflow.dflow DEBUG: Task 968 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/80e02ab6-9af0-43c8-bc04-b8094f8f4e59_calibrate_898695_36.stdout parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/80e02ab6-9af0-43c8-bc04-b8094f8f4e59_calibrate_898695_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:29 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 1019 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 180a92dfbc5dc0cb064f24fe3a7d37b1 with result from task 1019 parsl.dataflow.dflow DEBUG: Task 1019 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/6888f02d-4624-4cd7-94db-d62b259d20bb_calibrate_898800_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/6888f02d-4624-4cd7-94db-d62b259d20bb_calibrate_898800_22.stderr parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc8a411efb8421bb3a4d4b472c7ecd47 with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/8e28a9b9-03a1-48a3-888d-16b9f0042a28_calibrate_898798_10.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/8e28a9b9-03a1-48a3-888d-16b9f0042a28_calibrate_898798_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:29 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4cfb237232501a78b32cda3ef38321d with result from task 542 parsl.dataflow.dflow DEBUG: Task 542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/ec1f83d5-a716-4bde-b330-141853d2cf87_calibrate_898800_20.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/ec1f83d5-a716-4bde-b330-141853d2cf87_calibrate_898800_20.stderr parsl.dataflow.dflow INFO: Task 587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5d56c2ac32fdffb3832a8b90525a8e7 with result from task 587 parsl.dataflow.dflow DEBUG: Task 587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/1f6c643d-e470-492c-a047-ab8944a69dbe_calibrate_898799_29.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/1f6c643d-e470-492c-a047-ab8944a69dbe_calibrate_898799_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:30 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a8dbbcf72b10d9908c52f25ab3f260c with result from task 419 parsl.dataflow.dflow DEBUG: Task 419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/dc07d470-3104-48d3-aafa-0230005e358c_calibrate_898800_35.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/dc07d470-3104-48d3-aafa-0230005e358c_calibrate_898800_35.stderr parsl.dataflow.dflow INFO: Task 974 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1deab933c93afc3da0a31590d4a0f6b with result from task 974 parsl.dataflow.dflow DEBUG: Task 974 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/40b03996-a623-46aa-92ab-960c92c0c6cb_calibrate_898696_15.stdout parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/40b03996-a623-46aa-92ab-960c92c0c6cb_calibrate_898696_15.stderr parsl.dataflow.dflow INFO: Task 599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7828676b8f629a9f76954d6cebb90536 with result from task 599 parsl.dataflow.dflow DEBUG: Task 599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/8426e801-03e9-4255-8faa-7ca8d11f43ca_calibrate_898799_41.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/8426e801-03e9-4255-8faa-7ca8d11f43ca_calibrate_898799_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:30 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 845 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 792a7980eecd1d5aa4b66c5e5bd611b4 with result from task 845 parsl.dataflow.dflow DEBUG: Task 845 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/336c7a68-0841-4269-a5b4-6348483c723b_calibrate_898798_42.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/336c7a68-0841-4269-a5b4-6348483c723b_calibrate_898798_42.stderr parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b653a00884bcd82be18884cb1ea241a with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/c4ca6878-008d-479a-ae80-243e30837d7e_calibrate_898799_25.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/c4ca6878-008d-479a-ae80-243e30837d7e_calibrate_898799_25.stderr parsl.dataflow.dflow INFO: Task 698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e465d334b4a6db9696a64313c531a51 with result from task 698 parsl.dataflow.dflow DEBUG: Task 698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/4a2f70a0-ae2e-4105-8332-4e4906775c57_calibrate_898800_52.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/4a2f70a0-ae2e-4105-8332-4e4906775c57_calibrate_898800_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:31 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 734 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e827f8f2a67e597a7e8a6c3afe4dc8e1 with result from task 734 parsl.dataflow.dflow DEBUG: Task 734 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/353b9769-4f56-42d2-afa1-8ac1e4ab51c8_calibrate_898696_8.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/353b9769-4f56-42d2-afa1-8ac1e4ab51c8_calibrate_898696_8.stderr parsl.dataflow.dflow INFO: Task 797 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9ed2aaa60daaea70de57cb638421589 with result from task 797 parsl.dataflow.dflow DEBUG: Task 797 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/d5f5e494-ab7c-4c87-9683-1f7330397e0e_calibrate_898798_36.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/d5f5e494-ab7c-4c87-9683-1f7330397e0e_calibrate_898798_36.stderr parsl.dataflow.dflow INFO: Task 812 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f27fa66e30dcf1718c5c530620b1cfb0 with result from task 812 parsl.dataflow.dflow DEBUG: Task 812 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/22c4845a-52b5-4030-830e-838675df6f02_calibrate_898799_17.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/22c4845a-52b5-4030-830e-838675df6f02_calibrate_898799_17.stderr parsl.dataflow.dflow INFO: Task 895 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa97e55f61023b971683fcab46ce195f with result from task 895 parsl.dataflow.dflow DEBUG: Task 895 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 896 has memoization hash f1493b3ab8457afb2adeb65aca84a2fa parsl.dataflow.memoization INFO: Task 896 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c443920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 896 try 0 launched on executor multi with executor id 1080 parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/2d398caf-bc3b-43e1-9d65-d447b12d8f47_calibrate_898798_34.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/2d398caf-bc3b-43e1-9d65-d447b12d8f47_calibrate_898798_34.stderr parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/895d094b-d305-488b-8403-17510bdd1469_characterizeImage_898798_34.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/895d094b-d305-488b-8403-17510bdd1469_characterizeImage_898798_34.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd88c9360c509fa50b01674b3d4029d9 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/185f8949-b778-4aea-b0b1-da124503f21d_calibrate_898798_53.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/185f8949-b778-4aea-b0b1-da124503f21d_calibrate_898798_53.stderr parsl.dataflow.dflow INFO: Task 854 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e70f3ee4b8826b95a3dbd2146260ebd8 with result from task 854 parsl.dataflow.dflow DEBUG: Task 854 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/540f691f-4209-4e06-9645-f78dab5edca9_calibrate_898694_8.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/540f691f-4209-4e06-9645-f78dab5edca9_calibrate_898694_8.stderr parsl.dataflow.dflow INFO: Task 422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 321c769a36239c9953dc8ddf37abdcfd with result from task 422 parsl.dataflow.dflow DEBUG: Task 422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/18d9b4dd-c723-407d-9999-41d8481581dd_calibrate_898798_9.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/18d9b4dd-c723-407d-9999-41d8481581dd_calibrate_898798_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:31 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fc50da008ca41f1647693b9e34874e7 with result from task 476 parsl.dataflow.dflow DEBUG: Task 476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/19bf2d7c-b097-4c8b-8b3d-56ff61deb672_calibrate_898800_48.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/19bf2d7c-b097-4c8b-8b3d-56ff61deb672_calibrate_898800_48.stderr parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa6d122ba3ae916b25edc560410df878 with result from task 407 parsl.dataflow.dflow DEBUG: Task 407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/ca71a445-62c2-4a31-bfaf-3ab133f3dddc_calibrate_898800_25.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/ca71a445-62c2-4a31-bfaf-3ab133f3dddc_calibrate_898800_25.stderr parsl.dataflow.dflow INFO: Task 1025 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 220659b91ddc66d005a526aa72ee62dd with result from task 1025 parsl.dataflow.dflow DEBUG: Task 1025 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/4f3aa110-e615-47f8-8970-77fa3a8f4e6d_calibrate_898800_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/4f3aa110-e615-47f8-8970-77fa3a8f4e6d_calibrate_898800_6.stderr parsl.dataflow.dflow INFO: Task 962 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fa76147d5f4e98df58f18cc9e068894 with result from task 962 parsl.dataflow.dflow DEBUG: Task 962 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/d4c3bb47-7d85-4981-910f-36e5917bdb47_calibrate_898800_3.stdout parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/d4c3bb47-7d85-4981-910f-36e5917bdb47_calibrate_898800_3.stderr parsl.dataflow.dflow INFO: Task 1030 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca17b060ce9bf8cafc1fdc39c43368ff with result from task 1030 parsl.dataflow.dflow DEBUG: Task 1030 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1031 has memoization hash eee636e5a610172b9fa466856bd7da3f parsl.dataflow.memoization INFO: Task 1031 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1031 try 0 launched on executor multi with executor id 1081 parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/94e18b4e-e267-4367-ae8b-8b0cce1b56a9_calibrate_898798_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/94e18b4e-e267-4367-ae8b-8b0cce1b56a9_calibrate_898798_14.stderr parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/e4ac50b5-703e-4586-9f02-455fb7be6f7d_characterizeImage_898798_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/e4ac50b5-703e-4586-9f02-455fb7be6f7d_characterizeImage_898798_14.stderr parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9f825e3b1483d32f51c22c3c93e7586 with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash 0faaf19ab4fe8eac70147ce28343af19 parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 1082 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/27bffa7a-dd84-443c-a725-07fa7c14786f_calibrate_898799_15.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/27bffa7a-dd84-443c-a725-07fa7c14786f_calibrate_898799_15.stderr parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/b047e463-5d61-4e98-9862-f6a0706baec7_characterizeImage_898799_15.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/b047e463-5d61-4e98-9862-f6a0706baec7_characterizeImage_898799_15.stderr parsl.dataflow.dflow INFO: Task 826 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93f8fc1681c49e3d512a8deb210def6e with result from task 826 parsl.dataflow.dflow DEBUG: Task 826 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 827 has memoization hash 0e87d7acabf41234e5eeffbf717d71bc parsl.dataflow.memoization INFO: Task 827 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 827 try 0 launched on executor multi with executor id 1083 parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/019ef5cc-a130-4bdc-b3bb-857148877d9b_calibrate_898696_39.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/019ef5cc-a130-4bdc-b3bb-857148877d9b_calibrate_898696_39.stderr parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/9366a3d6-bcbb-43e6-8183-e796da08819d_characterizeImage_898696_39.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/9366a3d6-bcbb-43e6-8183-e796da08819d_characterizeImage_898696_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:32 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b44d913423193da1a4e21c4061e37169 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/be21b31b-6fde-401c-bdcb-47b1f96a4392_calibrate_898799_49.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/be21b31b-6fde-401c-bdcb-47b1f96a4392_calibrate_898799_49.stderr parsl.dataflow.dflow INFO: Task 881 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90e3028ef7699cb4288b35bf55a097bb with result from task 881 parsl.dataflow.dflow DEBUG: Task 881 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/44226f37-12be-4bf5-b6c4-1fa1273ebff5_calibrate_898798_3.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/44226f37-12be-4bf5-b6c4-1fa1273ebff5_calibrate_898798_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:32 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 1048 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fb7e5cd388c9d2e838752384499453c with result from task 1048 parsl.dataflow.dflow DEBUG: Task 1048 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1049 has memoization hash fa0d80f4ce9daa6228f09bed7678c664 parsl.dataflow.memoization INFO: Task 1049 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1049 try 0 launched on executor multi with executor id 1084 parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/a12bb995-b092-4f05-b853-f5bdf4ca8032_calibrate_898696_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/a12bb995-b092-4f05-b853-f5bdf4ca8032_calibrate_898696_9.stderr parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/46c31303-98e4-42ce-a32f-12c1908a604e_characterizeImage_898696_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/46c31303-98e4-42ce-a32f-12c1908a604e_characterizeImage_898696_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:33 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:33 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1166b738f333ba7560807fb2971a2f6 with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash e676e47f6f53df88d53e08ae13ee1a69 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d586480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 1085 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/51f75315-ba90-4346-9319-be973560e480_calibrate_898798_61.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/51f75315-ba90-4346-9319-be973560e480_calibrate_898798_61.stderr parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/0130ef3f-5e15-4576-ba86-2e69d69d6c09_characterizeImage_898798_61.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/0130ef3f-5e15-4576-ba86-2e69d69d6c09_characterizeImage_898798_61.stderr parsl.dataflow.dflow INFO: Task 998 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2f3dfae81c69787cec07fb135436569 with result from task 998 parsl.dataflow.dflow DEBUG: Task 998 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/cc9c34de-095b-4f01-a002-f5db16236fe2_calibrate_898798_19.stdout parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/cc9c34de-095b-4f01-a002-f5db16236fe2_calibrate_898798_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:34 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f0099ea4150e3abd38cb3d6bd6953de with result from task 550 parsl.dataflow.dflow DEBUG: Task 550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 551 has memoization hash 47f60ed3655f54901faf62a0ccd528d1 parsl.dataflow.memoization INFO: Task 551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dcea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 551 try 0 launched on executor multi with executor id 1086 parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/1cc45378-2909-42c9-aff1-a2dfe17cf3a4_calibrate_898799_28.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/1cc45378-2909-42c9-aff1-a2dfe17cf3a4_calibrate_898799_28.stderr parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/46f6f883-8578-4567-8689-8b699ed76217_characterizeImage_898799_28.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/46f6f883-8578-4567-8689-8b699ed76217_characterizeImage_898799_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:35 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4588dc4a22c0cbeee3ebeda0a46ee7a with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/42127a0b-b93d-4a2b-9fac-13f7c71ee2cf_calibrate_898798_32.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/42127a0b-b93d-4a2b-9fac-13f7c71ee2cf_calibrate_898798_32.stderr parsl.dataflow.dflow INFO: Task 947 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df4dc7fa77e9fe6cd034d3acd056afe3 with result from task 947 parsl.dataflow.dflow DEBUG: Task 947 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/10fa57a1-0bea-4738-9505-fcef5454b262_calibrate_898799_9.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/10fa57a1-0bea-4738-9505-fcef5454b262_calibrate_898799_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:35 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:36 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a016403407af9bed60019175f9e3c977 with result from task 497 parsl.dataflow.dflow DEBUG: Task 497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/dfc41065-6e5d-4754-b513-eb1b05a9713b_calibrate_898695_8.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/dfc41065-6e5d-4754-b513-eb1b05a9713b_calibrate_898695_8.stderr parsl.dataflow.dflow INFO: Task 878 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6608d008be45914b0b2c88e3fdf44617 with result from task 878 parsl.dataflow.dflow DEBUG: Task 878 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/3b4e0445-6ca8-47d5-8cc4-bb54ffb82ccc_calibrate_898696_25.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/3b4e0445-6ca8-47d5-8cc4-bb54ffb82ccc_calibrate_898696_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:36 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0ed6a4f6d4e1cb8f00f130549e69fc9 with result from task 437 parsl.dataflow.dflow DEBUG: Task 437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/b030deaa-e47d-4c51-8193-59016f884878_calibrate_898695_6.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/b030deaa-e47d-4c51-8193-59016f884878_calibrate_898695_6.stderr parsl.dataflow.dflow INFO: Task 1088 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3e31c999c23bad8c4a6abbd284126a9 with result from task 1088 parsl.dataflow.dflow DEBUG: Task 1088 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/bd8d2f44-8b63-4eff-a2e2-2523baadf0cc_calibrate_898800_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/bd8d2f44-8b63-4eff-a2e2-2523baadf0cc_calibrate_898800_16.stderr parsl.dataflow.dflow INFO: Task 1043 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a55a5da6afbf1784baece4e22fe8572 with result from task 1043 parsl.dataflow.dflow DEBUG: Task 1043 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/60947f7a-dae1-415d-a8bd-5653d541c143_calibrate_898798_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/60947f7a-dae1-415d-a8bd-5653d541c143_calibrate_898798_27.stderr parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c318de4e443a66efc243935240ed35a with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/79814f94-a4bc-4678-acb5-97c4b052ad04_calibrate_898798_17.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/79814f94-a4bc-4678-acb5-97c4b052ad04_calibrate_898798_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:37 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 926 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5874f13a53b1a8e303c1ebcb013d5b6b with result from task 926 parsl.dataflow.dflow DEBUG: Task 926 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/79f5e065-ac44-4bc8-87e2-8abfac2317c8_calibrate_898695_50.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/79f5e065-ac44-4bc8-87e2-8abfac2317c8_calibrate_898695_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:37 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91c82b481032f8f66dd610e10165a358 with result from task 554 parsl.dataflow.dflow DEBUG: Task 554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/328cd7f8-4999-4320-b70c-ed5701b4bbe2_calibrate_898800_29.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/328cd7f8-4999-4320-b70c-ed5701b4bbe2_calibrate_898800_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:38 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 1097 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cd46b9d2ae719cf1b0165b80d74be16 with result from task 1097 parsl.dataflow.dflow DEBUG: Task 1097 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/3ff5c90d-cecc-454c-813b-e659c8c8bf13_calibrate_898798_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/3ff5c90d-cecc-454c-813b-e659c8c8bf13_calibrate_898798_40.stderr parsl.dataflow.dflow INFO: Task 446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a15c16dd7b54435b74082aad18ca63c with result from task 446 parsl.dataflow.dflow DEBUG: Task 446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/bf9a525d-4ba7-43c7-b308-3dd1d938ace9_calibrate_898798_55.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/bf9a525d-4ba7-43c7-b308-3dd1d938ace9_calibrate_898798_55.stderr parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry daac6ad3e6e1762da6adbb28520bf2ca with result from task 425 parsl.dataflow.dflow DEBUG: Task 425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/776c9241-f0da-479c-b466-5a673a5faad6_calibrate_898798_30.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/776c9241-f0da-479c-b466-5a673a5faad6_calibrate_898798_30.stderr parsl.dataflow.dflow INFO: Task 461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e95f900783800f781da24f5e5a3fcc98 with result from task 461 parsl.dataflow.dflow DEBUG: Task 461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/2d17d49a-44d3-47d4-b792-8229ca2176de_calibrate_898695_11.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/2d17d49a-44d3-47d4-b792-8229ca2176de_calibrate_898695_11.stderr parsl.dataflow.dflow INFO: Task 932 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80e113d5e863b2ca6d6a05828a5565d5 with result from task 932 parsl.dataflow.dflow DEBUG: Task 932 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/2f904687-1650-4236-8999-3554b52dc5cd_calibrate_898694_9.stdout parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/2f904687-1650-4236-8999-3554b52dc5cd_calibrate_898694_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:39 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87ce344c6b4bd9be19c2f8c1fdd3beb2 with result from task 704 parsl.dataflow.dflow DEBUG: Task 704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/3e7c19dd-3837-4d47-a021-751233c28358_calibrate_898798_35.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/3e7c19dd-3837-4d47-a021-751233c28358_calibrate_898798_35.stderr parsl.dataflow.dflow INFO: Task 908 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eab6758fb96c928219c663d5db9d34ce with result from task 908 parsl.dataflow.dflow DEBUG: Task 908 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/1300a700-e5b4-447a-aecb-49e189df15df_calibrate_898696_3.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/1300a700-e5b4-447a-aecb-49e189df15df_calibrate_898696_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:39 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46414ece9bda60b78a8f5a9b2cfd27d1 with result from task 596 parsl.dataflow.dflow DEBUG: Task 596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/f672bb83-5a6d-4f52-8c3e-964482e9bfec_calibrate_898694_45.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/f672bb83-5a6d-4f52-8c3e-964482e9bfec_calibrate_898694_45.stderr parsl.dataflow.dflow INFO: Task 757 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 143dc0abcdff39d7b5871741e898b92e with result from task 757 parsl.dataflow.dflow DEBUG: Task 757 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 758 has memoization hash 9f8f17496b42643e85ef45bcb62d0795 parsl.dataflow.memoization INFO: Task 758 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3ba7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 758 try 0 launched on executor multi with executor id 1087 parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/baea2b06-7775-45ae-a999-35514ee8a027_calibrate_898800_39.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/baea2b06-7775-45ae-a999-35514ee8a027_calibrate_898800_39.stderr parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/626adb04-9592-43bb-bef1-50faf374065d_characterizeImage_898800_39.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/626adb04-9592-43bb-bef1-50faf374065d_characterizeImage_898800_39.stderr parsl.dataflow.dflow INFO: Task 518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5d1fa43497dcb64d8f106d6e1f70be8 with result from task 518 parsl.dataflow.dflow DEBUG: Task 518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/d30afcb6-106a-4009-93c7-a6cd3d5c7733_calibrate_898695_56.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/d30afcb6-106a-4009-93c7-a6cd3d5c7733_calibrate_898695_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:40 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 689 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f57fa9eedb88facc543594c7bc53d164 with result from task 689 parsl.dataflow.dflow DEBUG: Task 689 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/8b131649-f1ab-413a-a521-760198806f61_calibrate_898798_29.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/8b131649-f1ab-413a-a521-760198806f61_calibrate_898798_29.stderr parsl.dataflow.dflow INFO: Task 395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 327929318c9736360efc694f40cf2129 with result from task 395 parsl.dataflow.dflow DEBUG: Task 395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/df949289-307f-47da-9926-9f43a104f0fd_calibrate_898799_6.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/df949289-307f-47da-9926-9f43a104f0fd_calibrate_898799_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:40 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:41 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 788 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af6f9c7b31924800887ed4321a9f201a with result from task 788 parsl.dataflow.dflow DEBUG: Task 788 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/08d03992-9eab-4679-b25b-c89b9b8d07a4_calibrate_898799_16.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/08d03992-9eab-4679-b25b-c89b9b8d07a4_calibrate_898799_16.stderr parsl.dataflow.dflow INFO: Task 713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f248efeaa5877dfe55409b8dcb7f6ec with result from task 713 parsl.dataflow.dflow DEBUG: Task 713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/1e8f1a12-8dba-4b2d-82d3-2735ec2cd507_calibrate_898798_5.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/1e8f1a12-8dba-4b2d-82d3-2735ec2cd507_calibrate_898798_5.stderr parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d11467cb47cd42a26a7f9dfce426b2ee with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/26ae9a68-c5a5-4f67-ae7e-4b9cbfd47a90_calibrate_898695_12.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/26ae9a68-c5a5-4f67-ae7e-4b9cbfd47a90_calibrate_898695_12.stderr parsl.dataflow.dflow INFO: Task 1109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abd6857da903d1811efc3b37dbcc5ced with result from task 1109 parsl.dataflow.dflow DEBUG: Task 1109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/dda55a02-f2d7-41f8-ab5e-e4a2dde8bcf0_calibrate_898695_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/dda55a02-f2d7-41f8-ab5e-e4a2dde8bcf0_calibrate_898695_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:41 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b98794a6d2eb1f1e9b63e967f2960c1 with result from task 670 parsl.dataflow.dflow DEBUG: Task 670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 671 has memoization hash 7c90f9fc95df0fbefeb3a1e1ff041311 parsl.dataflow.memoization INFO: Task 671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3744a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 671 try 0 launched on executor multi with executor id 1088 parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/b659ea7e-a4f3-460c-beaf-54412afb9cef_calibrate_898799_1.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/b659ea7e-a4f3-460c-beaf-54412afb9cef_calibrate_898799_1.stderr parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/0774fad2-e5d5-493c-9daf-1337a6f7e800_characterizeImage_898799_1.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/0774fad2-e5d5-493c-9daf-1337a6f7e800_characterizeImage_898799_1.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8798a4cee6c13eb8e62627c86f6f470b with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/6364a26d-39bf-4644-9119-3e5580e7684e_calibrate_898799_58.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/6364a26d-39bf-4644-9119-3e5580e7684e_calibrate_898799_58.stderr parsl.dataflow.dflow INFO: Task 1058 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e74c88e975c4db8e4402fa2c32d0657e with result from task 1058 parsl.dataflow.dflow DEBUG: Task 1058 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/9122fb1e-5828-46cd-b4f4-b9acab8e0d18_calibrate_898798_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/9122fb1e-5828-46cd-b4f4-b9acab8e0d18_calibrate_898798_48.stderr parsl.dataflow.dflow INFO: Task 1033 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ffc3addcd1ca03a85bad88fadfb6d2f with result from task 1033 parsl.dataflow.dflow DEBUG: Task 1033 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1034 has memoization hash 277b18ea60bd8275363a54827f76aba4 parsl.dataflow.memoization INFO: Task 1034 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1034 try 0 launched on executor multi with executor id 1089 parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/bfec6a87-2d11-42ee-a7bb-cc8ec5902e01_calibrate_898798_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/bfec6a87-2d11-42ee-a7bb-cc8ec5902e01_calibrate_898798_26.stderr parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/b4850d3d-715c-4d24-9000-9e47d05da923_characterizeImage_898798_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/b4850d3d-715c-4d24-9000-9e47d05da923_characterizeImage_898798_26.stderr parsl.dataflow.dflow INFO: Task 611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da6ac486550e063297505f03dd83f23d with result from task 611 parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 611 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/cc55d994-914d-4b84-914c-b0270c3f48fc_calibrate_898696_60.stdout parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/cc55d994-914d-4b84-914c-b0270c3f48fc_calibrate_898696_60.stderr parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:42 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 1036 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0871e81032c443d811cc7355f982f04 with result from task 1036 parsl.dataflow.dflow DEBUG: Task 1036 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1037 has memoization hash fdcc231343b3e0b715a88543f43985a9 parsl.dataflow.memoization INFO: Task 1037 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1037 try 0 launched on executor multi with executor id 1090 parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/5c470043-e9d2-4070-9989-107fee30bea8_calibrate_898694_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/5c470043-e9d2-4070-9989-107fee30bea8_calibrate_898694_33.stderr parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/8fcf5d18-9ebc-419f-9aa5-de271b39895c_characterizeImage_898694_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/8fcf5d18-9ebc-419f-9aa5-de271b39895c_characterizeImage_898694_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:42 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 989 try 0 failed parsl.dataflow.dflow INFO: Task 989 marked for retry parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/69253044-9ab6-488f-b610-ee62d01a1524_calibrate_898695_61.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/69253044-9ab6-488f-b610-ee62d01a1524_calibrate_898695_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 989 has memoization hash 3dbddd233c41349e770215abb1f436ef parsl.dataflow.memoization INFO: Task 989 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 989 try 1 launched on executor multi with executor id 1091 parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/69253044-9ab6-488f-b610-ee62d01a1524_calibrate_898695_61.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/69253044-9ab6-488f-b610-ee62d01a1524_calibrate_898695_61.stderr parsl.dataflow.dflow INFO: Task 569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6ad85185a1a41dfd919473bcf921f26 with result from task 569 parsl.dataflow.dflow DEBUG: Task 569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/38cbf436-d110-471c-bf4d-8c04d4e37cc7_calibrate_898800_49.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/38cbf436-d110-471c-bf4d-8c04d4e37cc7_calibrate_898800_49.stderr parsl.dataflow.dflow INFO: Task 1091 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 295d67a9fdf024c2433fb435d7d83f2f with result from task 1091 parsl.dataflow.dflow DEBUG: Task 1091 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/9b6c248c-925c-46ec-b79c-e959b02f8a31_calibrate_898696_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/9b6c248c-925c-46ec-b79c-e959b02f8a31_calibrate_898696_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:43 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 970 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9f5ddc432108a5001d3a1c1f8aacdcc with result from task 970 parsl.dataflow.dflow DEBUG: Task 970 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 971 has memoization hash 7b5efbbe67fa224c4d17394255daa8ee parsl.dataflow.memoization INFO: Task 971 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 971 try 0 launched on executor multi with executor id 1092 parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/5a0e101b-8283-4b84-9b9b-f7e3dd3fa4f0_calibrate_898695_20.stdout parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/5a0e101b-8283-4b84-9b9b-f7e3dd3fa4f0_calibrate_898695_20.stderr parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/38f14a6f-d695-4a47-bb49-fc6ceb937657_characterizeImage_898695_20.stdout parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/38f14a6f-d695-4a47-bb49-fc6ceb937657_characterizeImage_898695_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:43 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 914 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 794bc8ba96dad960ab06a2fd489739de with result from task 914 parsl.dataflow.dflow DEBUG: Task 914 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/3ff78c46-0340-4b04-9f2d-23a418a9441f_calibrate_898799_12.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/3ff78c46-0340-4b04-9f2d-23a418a9441f_calibrate_898799_12.stderr parsl.dataflow.dflow INFO: Task 824 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df98eea0fead7f300e70067a3c1749c6 with result from task 824 parsl.dataflow.dflow DEBUG: Task 824 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/8af5af83-38df-4683-8032-095ce0c2ae9f_calibrate_898800_17.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/8af5af83-38df-4683-8032-095ce0c2ae9f_calibrate_898800_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:44 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e887dd1842e3a6bc8e7b50a982fd4a5 with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/1c0b1cbd-4917-44b4-b859-d2b9059cf389_calibrate_898800_46.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/1c0b1cbd-4917-44b4-b859-d2b9059cf389_calibrate_898800_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:44 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:45 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc39a5c25bc21ebe633e0d347ff0978e with result from task 584 parsl.dataflow.dflow DEBUG: Task 584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/0091b1a4-1eb7-416f-ab47-ff7cb8b00c68_calibrate_898695_21.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/0091b1a4-1eb7-416f-ab47-ff7cb8b00c68_calibrate_898695_21.stderr parsl.dataflow.dflow INFO: Task 674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a720fc71ca2f3943992ac246df8c3148 with result from task 674 parsl.dataflow.dflow DEBUG: Task 674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/13b6479c-a4fe-4bb5-ae3b-af49bfee5770_calibrate_898694_43.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/13b6479c-a4fe-4bb5-ae3b-af49bfee5770_calibrate_898694_43.stderr parsl.dataflow.dflow INFO: Task 752 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52493c34337ce01b709ec8c0a19c016a with result from task 752 parsl.dataflow.dflow DEBUG: Task 752 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/70d59114-f691-42f4-87a2-1c36f7110cba_calibrate_898800_43.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/70d59114-f691-42f4-87a2-1c36f7110cba_calibrate_898800_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:45 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4e2ac7d4d68fed80f3892b6e007b42f with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/25fa34c1-4166-4fe3-a2c5-e5180e360a6e_calibrate_898799_37.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/25fa34c1-4166-4fe3-a2c5-e5180e360a6e_calibrate_898799_37.stderr parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc8119de51d8f0290dd24af938480c9c with result from task 428 parsl.dataflow.dflow DEBUG: Task 428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/4496b301-6a96-4d87-b526-98a35a5fa0ca_calibrate_898799_19.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/4496b301-6a96-4d87-b526-98a35a5fa0ca_calibrate_898799_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:46 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd4f31959490e07b744a86a1d4d0e6cf with result from task 614 parsl.dataflow.dflow DEBUG: Task 614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/9c20547e-31a6-42ef-addf-9bf691544836_calibrate_898800_30.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/9c20547e-31a6-42ef-addf-9bf691544836_calibrate_898800_30.stderr parsl.dataflow.dflow INFO: Task 1073 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e68dffca902b875f2d6da9518a25b042 with result from task 1073 parsl.dataflow.dflow DEBUG: Task 1073 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/940e8366-0bff-4953-8982-e6472d9d702e_calibrate_898799_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/940e8366-0bff-4953-8982-e6472d9d702e_calibrate_898799_43.stderr parsl.dataflow.dflow INFO: Task 593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3c5fd385ebd7d3cd46d29c43f77967c with result from task 593 parsl.dataflow.dflow DEBUG: Task 593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/7345660d-bd61-450f-80b7-4f3de3486cd9_calibrate_898798_25.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/7345660d-bd61-450f-80b7-4f3de3486cd9_calibrate_898798_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:46 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:47 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 764 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 743850d95ab1218d0c8e881f600f6cd5 with result from task 764 parsl.dataflow.dflow DEBUG: Task 764 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/705820fd-e08c-4cbd-b3d7-96fff3f681bd_calibrate_898798_49.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/705820fd-e08c-4cbd-b3d7-96fff3f681bd_calibrate_898798_49.stderr parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b24c45cefb2392d628dc6a519e69873e with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/8b476ca5-2c03-479d-a372-df1adce0b6c5_calibrate_898695_39.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/8b476ca5-2c03-479d-a372-df1adce0b6c5_calibrate_898695_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:47 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6026b06d6c309cea767989e82e3503ae with result from task 500 parsl.dataflow.dflow DEBUG: Task 500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/bb62a931-b47d-4408-89b1-61ff44a856e1_calibrate_898799_30.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/bb62a931-b47d-4408-89b1-61ff44a856e1_calibrate_898799_30.stderr parsl.dataflow.dflow INFO: Task 539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 050cef99d055545deae8e5edf1d05a23 with result from task 539 parsl.dataflow.dflow DEBUG: Task 539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/75e066f3-cb9a-4348-b558-62fbb8c110c9_calibrate_898800_32.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/75e066f3-cb9a-4348-b558-62fbb8c110c9_calibrate_898800_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 83 parsl.dataflow.strategy DEBUG: Executor multi has 83 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 2024-03-13 09:49:48 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local parsl.dataflow.dflow INFO: Task 782 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d137f99a14c7f08c6cabd8de46f218ed with result from task 782 parsl.dataflow.dflow DEBUG: Task 782 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/1755b4e3-7489-4115-98a7-516fbb481437_calibrate_898800_28.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/1755b4e3-7489-4115-98a7-516fbb481437_calibrate_898800_28.stderr 2024-03-13 09:49:48 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 581bdfa90b2f6e9adfbc9da78cfbf99a with result from task 680 parsl.dataflow.dflow DEBUG: Task 680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/76c675ff-d57e-46a8-ade1-83abc1ccfb98_calibrate_898696_44.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/76c675ff-d57e-46a8-ade1-83abc1ccfb98_calibrate_898696_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:49 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cfd6e53a2e842a19a45897acd24dfaf with result from task 323 parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/93e475bd-638b-4b75-a5fa-4d0ca168b5bb_calibrate_898695_19.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/93e475bd-638b-4b75-a5fa-4d0ca168b5bb_calibrate_898695_19.stderr parsl.dataflow.dflow INFO: Task 701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2da480e23dd165e01ae48c23d7845006 with result from task 701 parsl.dataflow.dflow DEBUG: Task 701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/268a55aa-3a60-4331-921e-ab939327c69f_calibrate_898694_19.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/268a55aa-3a60-4331-921e-ab939327c69f_calibrate_898694_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:49 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5eba159c48b877decafcd6876bde76f with result from task 478 parsl.dataflow.dflow DEBUG: Task 478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 479 has memoization hash 8963f8b5afe42643d5b20b3f74fbcf0f parsl.dataflow.memoization INFO: Task 479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c482ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 1093 parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/479fd069-8081-407e-9a79-44a06c69b06f_calibrate_898798_28.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/479fd069-8081-407e-9a79-44a06c69b06f_calibrate_898798_28.stderr parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/54651e0d-ed4a-4b9c-b1f0-273dedbbc000_characterizeImage_898798_28.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/54651e0d-ed4a-4b9c-b1f0-273dedbbc000_characterizeImage_898798_28.stderr parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3d1d7455bc6f61b1caf63874d1a6e4c with result from task 344 parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/772d0c16-9e08-4c35-9084-0975b5db18a7_calibrate_898695_54.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/772d0c16-9e08-4c35-9084-0975b5db18a7_calibrate_898695_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:49 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d05c08ea51a30ad4eb2185abd081efd with result from task 563 parsl.dataflow.dflow DEBUG: Task 563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/a5929016-1a94-46b4-a7b2-79d74e537e2b_calibrate_898694_21.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/a5929016-1a94-46b4-a7b2-79d74e537e2b_calibrate_898694_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:50 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:50 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 953 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 248e60da8932eb6d81d61195d1af172c with result from task 953 parsl.dataflow.dflow DEBUG: Task 953 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/65bc30d4-91d8-465b-bd8e-d208c112fdc2_calibrate_898798_16.stdout parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/65bc30d4-91d8-465b-bd8e-d208c112fdc2_calibrate_898798_16.stderr parsl.dataflow.dflow INFO: Task 685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52b97924429232f45030d5ee95a3ef91 with result from task 685 parsl.dataflow.dflow DEBUG: Task 685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 686 has memoization hash ea36cfc5619ef9511f17b81f0cfad3d7 parsl.dataflow.memoization INFO: Task 686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3754e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 686 try 0 launched on executor multi with executor id 1094 parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/2bdc3116-8489-448a-a7be-ee983079368b_calibrate_898799_50.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/2bdc3116-8489-448a-a7be-ee983079368b_calibrate_898799_50.stderr parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/7b3cd1d6-c316-4af7-bd9c-0494d222cc41_characterizeImage_898799_50.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/7b3cd1d6-c316-4af7-bd9c-0494d222cc41_characterizeImage_898799_50.stderr parsl.dataflow.dflow INFO: Task 637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6565e32a853e0577e8ffe73abc395ab with result from task 637 parsl.dataflow.dflow DEBUG: Task 637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 638 has memoization hash 6951be3cbe1ad3c652e7cb688acbf1e2 parsl.dataflow.memoization INFO: Task 638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c512b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 638 try 0 launched on executor multi with executor id 1095 parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/9cb84eca-b1f7-42d0-b66e-369ec6988137_calibrate_898799_13.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/9cb84eca-b1f7-42d0-b66e-369ec6988137_calibrate_898799_13.stderr parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/c1adc23c-864b-4eb7-96d3-a034c1d7018c_characterizeImage_898799_13.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/c1adc23c-864b-4eb7-96d3-a034c1d7018c_characterizeImage_898799_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:51 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34d44fe7d331f8695ba08a20628c97cd with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash c2e255a54b7676c2635aa6df65a89f47 parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d660a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 1096 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/2b15710e-26b2-4833-bace-4bb7c0d8ac51_calibrate_898695_49.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/2b15710e-26b2-4833-bace-4bb7c0d8ac51_calibrate_898695_49.stderr parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/28b92bf0-1493-4b2a-bf77-8b7085c9a03a_characterizeImage_898695_49.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/28b92bf0-1493-4b2a-bf77-8b7085c9a03a_characterizeImage_898695_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:51 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f8b1ca3d1f9f8ad7f99f751c7767895 with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash ae2d37ce7ae63d7dac4a4330499b5f3a parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d660400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 1097 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/2fb6fa88-9349-4ec5-8347-4e6637291066_calibrate_898694_56.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/2fb6fa88-9349-4ec5-8347-4e6637291066_calibrate_898694_56.stderr parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/38f5c59d-3926-42a3-9703-846d89729d50_characterizeImage_898694_56.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/38f5c59d-3926-42a3-9703-846d89729d50_characterizeImage_898694_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:52 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 779 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b32452c53097f058ad9910cb8f34a9a6 with result from task 779 parsl.dataflow.dflow DEBUG: Task 779 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/dadc5136-9729-4a59-8897-547da1780bd3_calibrate_898800_13.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/dadc5136-9729-4a59-8897-547da1780bd3_calibrate_898800_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:52 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 755 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71c1df5265c6a7e5d73bca960a81b496 with result from task 755 parsl.dataflow.dflow DEBUG: Task 755 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/e139a0dc-d854-4395-9af3-1a5852ba4a41_calibrate_898694_20.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/e139a0dc-d854-4395-9af3-1a5852ba4a41_calibrate_898694_20.stderr parsl.dataflow.dflow INFO: Task 1028 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2461f9ea5f7bf6f066ce2e7c9d093fa4 with result from task 1028 parsl.dataflow.dflow DEBUG: Task 1028 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/d81ff872-acd5-4a20-b0d6-3f1de3120f30_calibrate_898694_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/d81ff872-acd5-4a20-b0d6-3f1de3120f30_calibrate_898694_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:53 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 917 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0ce5d3a8dea693e786b6eb550160019 with result from task 917 parsl.dataflow.dflow DEBUG: Task 917 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/726a9c2c-16a3-47a4-b083-ffb148b41bce_calibrate_898800_14.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/726a9c2c-16a3-47a4-b083-ffb148b41bce_calibrate_898800_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:53 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fea4ca5152b8eac8a5866c2f4332ecdd with result from task 544 parsl.dataflow.dflow DEBUG: Task 544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 545 has memoization hash 3181f7eb15308993c8ce269c97aebbcf parsl.dataflow.memoization INFO: Task 545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dc860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 1098 parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/8bce8e9a-1e2c-4a22-9eb5-ba58485bace3_calibrate_898799_48.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/8bce8e9a-1e2c-4a22-9eb5-ba58485bace3_calibrate_898799_48.stderr parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/71a9e1af-f47b-4d79-b7e2-23e2eef24fdd_characterizeImage_898799_48.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/71a9e1af-f47b-4d79-b7e2-23e2eef24fdd_characterizeImage_898799_48.stderr parsl.dataflow.dflow INFO: Task 883 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89322ae988c8e38d298f4f6f11f01cd7 with result from task 883 parsl.dataflow.dflow DEBUG: Task 883 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 884 has memoization hash acba211c60af95f87cd6741715c11d03 parsl.dataflow.memoization INFO: Task 884 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c442ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 884 try 0 launched on executor multi with executor id 1099 parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/5a2103a6-f3f4-4829-bdff-b05c0288a4d3_calibrate_898798_22.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/5a2103a6-f3f4-4829-bdff-b05c0288a4d3_calibrate_898798_22.stderr parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/e157aee2-b62e-4d1a-a5fb-6c0c83755762_characterizeImage_898798_22.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/e157aee2-b62e-4d1a-a5fb-6c0c83755762_characterizeImage_898798_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:53 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 794 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b38e5a00df0a7a72bdfb375e5d5e0ba with result from task 794 parsl.dataflow.dflow DEBUG: Task 794 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/4fdf0481-6645-43fc-b4ed-5286b4e9678d_calibrate_898800_44.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/4fdf0481-6645-43fc-b4ed-5286b4e9678d_calibrate_898800_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:54 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ac4f5c2bf063a5efa2692a0b0853b14 with result from task 590 parsl.dataflow.dflow DEBUG: Task 590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/32aa74c9-e928-4db8-9593-f9165e0185f5_calibrate_898695_24.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/32aa74c9-e928-4db8-9593-f9165e0185f5_calibrate_898695_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:54 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:55 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 776 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e782f0819ec666a0a24cfdb39a2a9b10 with result from task 776 parsl.dataflow.dflow DEBUG: Task 776 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/6fa1469d-f3c8-4445-bbf8-d9611363b150_calibrate_898694_4.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/6fa1469d-f3c8-4445-bbf8-d9611363b150_calibrate_898694_4.stderr parsl.dataflow.dflow INFO: Task 452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 750feeb3e623ed64f7c0e47fe20cc744 with result from task 452 parsl.dataflow.dflow DEBUG: Task 452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/cbc0b4f3-0d84-4cc7-a1fa-c863c64cfd76_calibrate_898799_60.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/cbc0b4f3-0d84-4cc7-a1fa-c863c64cfd76_calibrate_898799_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:55 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 839 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b216891f4f71de97dd4c4eb7359c666 with result from task 839 parsl.dataflow.dflow DEBUG: Task 839 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/f572d6d7-7f23-4d74-bd66-bf8674cec333_calibrate_898798_44.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/f572d6d7-7f23-4d74-bd66-bf8674cec333_calibrate_898798_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:56 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:56 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 821 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39a8d98c4e54ff2b9cafee43c51a4baf with result from task 821 parsl.dataflow.dflow DEBUG: Task 821 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/cadb0179-1aed-4e65-b6d8-4b309602fe2b_calibrate_898696_51.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/cadb0179-1aed-4e65-b6d8-4b309602fe2b_calibrate_898696_51.stderr parsl.dataflow.dflow INFO: Task 1076 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2aa6b814cae5b6cff944ebcddf7bc13 with result from task 1076 parsl.dataflow.dflow DEBUG: Task 1076 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/315e6269-759d-4f3e-950c-b000f419b0c5_calibrate_898695_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/315e6269-759d-4f3e-950c-b000f419b0c5_calibrate_898695_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:57 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3088e3bd967f064981ded5bd80f25d3 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/d2eb003e-0dc4-4adf-bbfa-ae866070e406_calibrate_898800_1.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/d2eb003e-0dc4-4adf-bbfa-ae866070e406_calibrate_898800_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:57 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:58 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e45f9f10518590763731cad502ec5d78 with result from task 374 parsl.dataflow.dflow DEBUG: Task 374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/8532ddc3-7d2a-410e-98d1-cff0f3cc0f7c_calibrate_898696_43.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/8532ddc3-7d2a-410e-98d1-cff0f3cc0f7c_calibrate_898696_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:58 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a20750f439edb585258e97d1175502f1 with result from task 601 parsl.dataflow.dflow DEBUG: Task 601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 602 has memoization hash 71e9480734f0caf45603fcd9340e5292 parsl.dataflow.memoization INFO: Task 602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c510220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 602 try 0 launched on executor multi with executor id 1100 parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/ade8159f-4e3e-4d16-b014-44cff63c7b89_calibrate_898800_23.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/ade8159f-4e3e-4d16-b014-44cff63c7b89_calibrate_898800_23.stderr parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/53ab60d2-2cf3-4c46-9374-602242198b08_characterizeImage_898800_23.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/53ab60d2-2cf3-4c46-9374-602242198b08_characterizeImage_898800_23.stderr parsl.dataflow.dflow INFO: Task 899 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 870b62109b471c6aa6253da49b8bf2a3 with result from task 899 parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.dataflow.dflow DEBUG: Task 899 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/af4a8e3f-f4a0-46e8-a513-52f595eedb2c_calibrate_898800_56.stdout parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/af4a8e3f-f4a0-46e8-a513-52f595eedb2c_calibrate_898800_56.stderr parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:59 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9fb7222e1834314661c00a410cc99b2 with result from task 377 parsl.dataflow.dflow DEBUG: Task 377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/add8b1de-a1e0-4886-9af8-aa236e1e9f0d_calibrate_898694_40.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/add8b1de-a1e0-4886-9af8-aa236e1e9f0d_calibrate_898694_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:59 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:49:59 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 815 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a91c910ab9f488017aeb13f5563cfbb with result from task 815 parsl.dataflow.dflow DEBUG: Task 815 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/a856c6b5-69ed-4363-8d66-9168615088e3_calibrate_898799_21.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/a856c6b5-69ed-4363-8d66-9168615088e3_calibrate_898799_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:00 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 956 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf04e937a3427fe98700f21a9c68f79c with result from task 956 parsl.dataflow.dflow DEBUG: Task 956 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/4fd6e5cb-c7e0-4b95-a610-d005ba44b67b_calibrate_898798_1.stdout parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/4fd6e5cb-c7e0-4b95-a610-d005ba44b67b_calibrate_898798_1.stderr parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c86735254fbf9f1bb8229c31ba15684 with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/dc438aad-ed44-4aa3-a905-514717db4f0a_calibrate_898695_27.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/dc438aad-ed44-4aa3-a905-514717db4f0a_calibrate_898695_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:00 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 415a9d92fc459545afdf36c7a6134347 with result from task 530 parsl.dataflow.dflow DEBUG: Task 530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/aa30f444-6777-4311-9acb-263508b44d0a_calibrate_898696_23.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/aa30f444-6777-4311-9acb-263508b44d0a_calibrate_898696_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:01 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:01 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69e6a497d3b0ee62425293a05ddfaf05 with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/18afa585-c8f2-40c1-82dc-2f662b67e345_calibrate_898800_60.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/18afa585-c8f2-40c1-82dc-2f662b67e345_calibrate_898800_60.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea6679d188b265f0d66a2b2fd5863c02 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/fd2b2284-dda2-4b03-9bd0-2adea089b3f4_calibrate_898798_46.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/fd2b2284-dda2-4b03-9bd0-2adea089b3f4_calibrate_898798_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:01 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:02 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.dataflow.dflow INFO: Task 1106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a788bb505a90dc25fc9b5b190c866ce6 with result from task 1106 parsl.dataflow.dflow DEBUG: Task 1106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/326eb58f-0b1d-496a-8037-d63a1cdfbcff_calibrate_898798_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/326eb58f-0b1d-496a-8037-d63a1cdfbcff_calibrate_898798_43.stderr parsl.dataflow.dflow INFO: Task 922 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a79a82c27da9bf8cec106e76321500ab with result from task 922 parsl.dataflow.dflow DEBUG: Task 922 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 923 has memoization hash c3da1c062b14f3a9eeabe4704c476f13 parsl.dataflow.memoization INFO: Task 923 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c293ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 923 try 0 launched on executor multi with executor id 1101 parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/0404a286-2598-424d-b0fe-943829e11c00_calibrate_898695_13.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/0404a286-2598-424d-b0fe-943829e11c00_calibrate_898695_13.stderr parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/58d9de2b-fffd-4eab-b61f-287c21bbadf7_characterizeImage_898695_13.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/58d9de2b-fffd-4eab-b61f-287c21bbadf7_characterizeImage_898695_13.stderr parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50c0c9f57577354a9fef86c07aaa78ba with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/5d9fb8c4-d209-4762-a403-ac39853f47d4_calibrate_898695_40.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/5d9fb8c4-d209-4762-a403-ac39853f47d4_calibrate_898695_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:02 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.dataflow.dflow INFO: Task 827 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e87d7acabf41234e5eeffbf717d71bc with result from task 827 parsl.dataflow.dflow DEBUG: Task 827 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/019ef5cc-a130-4bdc-b3bb-857148877d9b_calibrate_898696_39.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/019ef5cc-a130-4bdc-b3bb-857148877d9b_calibrate_898696_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:03 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.dataflow.dflow INFO: Task 692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa5b57bbbdccc25e47882d98dfe89330 with result from task 692 parsl.dataflow.dflow DEBUG: Task 692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/66882944-caeb-4f41-ac36-7157c5718603_calibrate_898695_34.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/66882944-caeb-4f41-ac36-7157c5718603_calibrate_898695_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:03 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.dataflow.dflow INFO: Task 830 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca8ec6159a3844b891fbb6bde691dea7 with result from task 830 parsl.dataflow.dflow DEBUG: Task 830 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/7c7bb6bc-14bd-47b3-b254-4fe50d508dd2_calibrate_898694_34.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/7c7bb6bc-14bd-47b3-b254-4fe50d508dd2_calibrate_898694_34.stderr parsl.dataflow.dflow INFO: Task 635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f018b220b7650b9eaf349698338e420 with result from task 635 parsl.dataflow.dflow DEBUG: Task 635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/c91421ed-2df9-4855-848b-5d5ed8e3c00e_calibrate_898798_45.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/c91421ed-2df9-4855-848b-5d5ed8e3c00e_calibrate_898798_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:03 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:04 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a847d9d9a6bb334b6fa1f06f1c6737cf with result from task 503 parsl.dataflow.dflow DEBUG: Task 503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/345aae2b-7b09-4229-b4ba-5cba7f076cd0_calibrate_898798_58.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/345aae2b-7b09-4229-b4ba-5cba7f076cd0_calibrate_898798_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:04 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 887 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24d8008216a733d30f5754c2aa671137 with result from task 887 parsl.dataflow.dflow DEBUG: Task 887 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/d88de9c2-212c-46e0-817e-b88fe3722b6e_calibrate_898799_14.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/d88de9c2-212c-46e0-817e-b88fe3722b6e_calibrate_898799_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:05 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:05 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:05 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d75758b6433e05d1ab259217bc2cde4 with result from task 448 parsl.dataflow.dflow DEBUG: Task 448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 449 has memoization hash 270994a4c6980a066f0e5170b4ed0c20 parsl.dataflow.memoization INFO: Task 449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4a1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 449 try 0 launched on executor multi with executor id 1102 parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/ba2dc6d5-7c01-4db0-bd32-cf8c7d18cde6_calibrate_898695_43.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/ba2dc6d5-7c01-4db0-bd32-cf8c7d18cde6_calibrate_898695_43.stderr parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/40fe8503-3ff8-4b5d-b5e1-a9455b6083fa_characterizeImage_898695_43.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/40fe8503-3ff8-4b5d-b5e1-a9455b6083fa_characterizeImage_898695_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:06 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:06 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:06 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 1049 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa0d80f4ce9daa6228f09bed7678c664 with result from task 1049 parsl.dataflow.dflow DEBUG: Task 1049 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/a12bb995-b092-4f05-b853-f5bdf4ca8032_calibrate_898696_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/a12bb995-b092-4f05-b853-f5bdf4ca8032_calibrate_898696_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:07 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11ddcafe2f5f9ee40faa626adb7e37f6 with result from task 521 parsl.dataflow.dflow DEBUG: Task 521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/6c9a0081-fd39-441b-b2f1-f4d4b497e8b4_calibrate_898799_44.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/6c9a0081-fd39-441b-b2f1-f4d4b497e8b4_calibrate_898799_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:07 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 920 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7739606b6d9e735452b63c5495b39d1 with result from task 920 parsl.dataflow.dflow DEBUG: Task 920 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/72117df8-53ff-49ac-bfb5-7725f8896c2d_calibrate_898800_5.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/72117df8-53ff-49ac-bfb5-7725f8896c2d_calibrate_898800_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:07 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:08 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:08 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:09 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:09 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:09 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 989 try 1 failed parsl.dataflow.dflow ERROR: Task 989 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 3dbddd233c41349e770215abb1f436ef with result from task 989 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/69253044-9ab6-488f-b610-ee62d01a1524_calibrate_898695_61.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/69253044-9ab6-488f-b610-ee62d01a1524_calibrate_898695_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:10 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:10 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3e3edd96503b13afda0b5a95429845d with result from task 677 parsl.dataflow.dflow DEBUG: Task 677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/d3088256-e68b-4fa6-ba5a-91bd5771a5e9_calibrate_898695_1.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/d3088256-e68b-4fa6-ba5a-91bd5771a5e9_calibrate_898695_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:10 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:11 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c74183afd69a4f46d7614b52ca098f6a with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/fba4e545-b55a-4abd-b6da-a231cf963a22_calibrate_898800_58.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/fba4e545-b55a-4abd-b6da-a231cf963a22_calibrate_898800_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:11 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:11 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:12 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 971 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b5efbbe67fa224c4d17394255daa8ee with result from task 971 parsl.dataflow.dflow DEBUG: Task 971 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/5a0e101b-8283-4b84-9b9b-f7e3dd3fa4f0_calibrate_898695_20.stdout parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/5a0e101b-8283-4b84-9b9b-f7e3dd3fa4f0_calibrate_898695_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:12 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 902 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9547a0d715b8dee055d6c9afe8bbbb2f with result from task 902 parsl.dataflow.dflow DEBUG: Task 902 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/c01c12ae-af7c-4fcd-8284-37c2b4804bf5_calibrate_898800_34.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/c01c12ae-af7c-4fcd-8284-37c2b4804bf5_calibrate_898800_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:12 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:13 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:13 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 1037 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdcc231343b3e0b715a88543f43985a9 with result from task 1037 parsl.dataflow.dflow DEBUG: Task 1037 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/5c470043-e9d2-4070-9989-107fee30bea8_calibrate_898694_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/5c470043-e9d2-4070-9989-107fee30bea8_calibrate_898694_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:13 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:14 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:14 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:14 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 544fb403a8fa0754c5da5c30fde66127 with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/ec251919-71d9-4b1d-8ead-98757f07921f_calibrate_898799_45.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/ec251919-71d9-4b1d-8ead-98757f07921f_calibrate_898799_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:15 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:15 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 1081 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a93fa3a7118ad475dc1948b41fafb175 with result from task 1081 parsl.dataflow.dflow DEBUG: Task 1081 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1082 has memoization hash a401c6c4baa9c5fd77b4a698de604828 parsl.dataflow.memoization INFO: Task 1082 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c329f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1082 try 0 launched on executor multi with executor id 1103 parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/e228f848-423f-4d58-aed4-04b5e94400b0_calibrate_898695_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/e228f848-423f-4d58-aed4-04b5e94400b0_calibrate_898695_23.stderr parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/53e233bf-39f2-44f2-816b-76732127389d_characterizeImage_898695_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/53e233bf-39f2-44f2-816b-76732127389d_characterizeImage_898695_23.stderr parsl.dataflow.dflow INFO: Task 580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6edfbaa8a43a6cc5b68e24ce80ec1bd with result from task 580 parsl.dataflow.dflow DEBUG: Task 580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 581 has memoization hash c74ba802dc820a19e170c65af1f4f9ed parsl.dataflow.memoization INFO: Task 581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4dea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 581 try 0 launched on executor multi with executor id 1104 parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/d4727172-c61d-4cd9-bc37-29eefcb6061e_calibrate_898798_60.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/d4727172-c61d-4cd9-bc37-29eefcb6061e_calibrate_898798_60.stderr parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/684e423b-7138-4544-92c6-f77f1e7817d1_characterizeImage_898798_60.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898798/684e423b-7138-4544-92c6-f77f1e7817d1_characterizeImage_898798_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:15 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:16 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:16 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:16 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 1031 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eee636e5a610172b9fa466856bd7da3f with result from task 1031 parsl.dataflow.dflow DEBUG: Task 1031 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/94e18b4e-e267-4367-ae8b-8b0cce1b56a9_calibrate_898798_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/94e18b4e-e267-4367-ae8b-8b0cce1b56a9_calibrate_898798_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:17 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:17 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:17 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:18 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:18 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:18 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:19 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0faaf19ab4fe8eac70147ce28343af19 with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/27bffa7a-dd84-443c-a725-07fa7c14786f_calibrate_898799_15.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/27bffa7a-dd84-443c-a725-07fa7c14786f_calibrate_898799_15.stderr parsl.dataflow.dflow INFO: Task 761 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 158011859d6499de22593cd90fd217a5 with result from task 761 parsl.dataflow.dflow DEBUG: Task 761 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/a4b1a521-e556-42cd-bffa-4b0454972074_calibrate_898799_38.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/a4b1a521-e556-42cd-bffa-4b0454972074_calibrate_898799_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:19 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2e255a54b7676c2635aa6df65a89f47 with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/2b15710e-26b2-4833-bace-4bb7c0d8ac51_calibrate_898695_49.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/2b15710e-26b2-4833-bace-4bb7c0d8ac51_calibrate_898695_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:20 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae2d37ce7ae63d7dac4a4330499b5f3a with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/2fb6fa88-9349-4ec5-8347-4e6637291066_calibrate_898694_56.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/2fb6fa88-9349-4ec5-8347-4e6637291066_calibrate_898694_56.stderr parsl.dataflow.dflow INFO: Task 551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47f60ed3655f54901faf62a0ccd528d1 with result from task 551 parsl.dataflow.dflow DEBUG: Task 551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/1cc45378-2909-42c9-aff1-a2dfe17cf3a4_calibrate_898799_28.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/1cc45378-2909-42c9-aff1-a2dfe17cf3a4_calibrate_898799_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:20 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 896 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1493b3ab8457afb2adeb65aca84a2fa with result from task 896 parsl.dataflow.dflow DEBUG: Task 896 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/2d398caf-bc3b-43e1-9d65-d447b12d8f47_calibrate_898798_34.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/2d398caf-bc3b-43e1-9d65-d447b12d8f47_calibrate_898798_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:20 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:21 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:21 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:21 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:21 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:22 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:22 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:22 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 431ee53e06ce383c7a6058b68c897eaf with result from task 631 parsl.dataflow.dflow DEBUG: Task 631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 632 has memoization hash f1ee3936b809e4027572c312171da12a parsl.dataflow.memoization INFO: Task 632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c512520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 632 try 0 launched on executor multi with executor id 1105 parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/c7523ad6-8291-4407-95d4-2d97b846ff1b_calibrate_898696_49.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/c7523ad6-8291-4407-95d4-2d97b846ff1b_calibrate_898696_49.stderr parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/b13f33ae-58f9-4457-b9c0-ab37d39c4266_characterizeImage_898696_49.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/b13f33ae-58f9-4457-b9c0-ab37d39c4266_characterizeImage_898696_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:23 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 30 parsl.dataflow.strategy DEBUG: Executor multi has 30 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 2024-03-13 09:50:23 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 09:50:23 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:23 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:24 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef67ae7a6b88d5b735e51b2ebd0fae4e with result from task 431 parsl.dataflow.dflow DEBUG: Task 431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/44842707-2af9-4f33-abb3-3487b385dc52_calibrate_898799_33.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/44842707-2af9-4f33-abb3-3487b385dc52_calibrate_898799_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:24 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:24 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:25 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:25 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:26 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:26 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:26 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:27 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:27 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:27 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:28 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:28 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:28 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 1034 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 277b18ea60bd8275363a54827f76aba4 with result from task 1034 parsl.dataflow.dflow DEBUG: Task 1034 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/bfec6a87-2d11-42ee-a7bb-cc8ec5902e01_calibrate_898798_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/bfec6a87-2d11-42ee-a7bb-cc8ec5902e01_calibrate_898798_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:29 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:29 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:29 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:30 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:30 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 758 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f8f17496b42643e85ef45bcb62d0795 with result from task 758 parsl.dataflow.dflow DEBUG: Task 758 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/baea2b06-7775-45ae-a999-35514ee8a027_calibrate_898800_39.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/baea2b06-7775-45ae-a999-35514ee8a027_calibrate_898800_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:30 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:31 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:31 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:31 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:32 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:32 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:32 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea36cfc5619ef9511f17b81f0cfad3d7 with result from task 686 parsl.dataflow.dflow DEBUG: Task 686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/2bdc3116-8489-448a-a7be-ee983079368b_calibrate_898799_50.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/2bdc3116-8489-448a-a7be-ee983079368b_calibrate_898799_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:33 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:33 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:33 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:34 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.dataflow.dflow INFO: Task 638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6951be3cbe1ad3c652e7cb688acbf1e2 with result from task 638 parsl.dataflow.dflow DEBUG: Task 638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/9cb84eca-b1f7-42d0-b66e-369ec6988137_calibrate_898799_13.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/9cb84eca-b1f7-42d0-b66e-369ec6988137_calibrate_898799_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:34 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:34 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:35 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:35 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 656 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5b7a3d3394e9b88c975445821d06028 with result from task 656 parsl.dataflow.dflow DEBUG: Task 656 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/327bce8a-87c7-4f8a-878c-1ba825a16f79_calibrate_898800_45.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/327bce8a-87c7-4f8a-878c-1ba825a16f79_calibrate_898800_45.stderr parsl.dataflow.dflow INFO: Task 910 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ea21cd80cc74b6680a5069b34058473 with result from task 910 parsl.dataflow.dflow DEBUG: Task 910 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 911 has memoization hash 24bcaaa7f86e276b47e47b8d207e2604 parsl.dataflow.memoization INFO: Task 911 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c290720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Parsl task 911 try 0 launched on executor multi with executor id 1106 parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/f208e14d-91f6-4f71-b325-767aeca7defc_calibrate_898800_4.stdout parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/f208e14d-91f6-4f71-b325-767aeca7defc_calibrate_898800_4.stderr parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/6e76686c-6764-440c-a9d6-f0fb71d2ec63_characterizeImage_898800_4.stdout parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:35 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/6e76686c-6764-440c-a9d6-f0fb71d2ec63_characterizeImage_898800_4.stderr proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:36 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:36 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:36 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:36 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:37 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:37 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:37 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:38 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:38 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:38 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:39 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:39 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 270994a4c6980a066f0e5170b4ed0c20 with result from task 449 parsl.dataflow.dflow DEBUG: Task 449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/ba2dc6d5-7c01-4db0-bd32-cf8c7d18cde6_calibrate_898695_43.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/ba2dc6d5-7c01-4db0-bd32-cf8c7d18cde6_calibrate_898695_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:39 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:40 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:40 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:40 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:41 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18b5e8fb531024c205678fef3cf5a45c with result from task 493 parsl.dataflow.dflow DEBUG: Task 493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 494 has memoization hash 890357a22897dc698e14a56947154fd7 parsl.dataflow.memoization INFO: Task 494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c4839c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 1107 parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/ad4849d7-78ca-4b1e-98c3-72f53ff79d8f_calibrate_898696_30.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/ad4849d7-78ca-4b1e-98c3-72f53ff79d8f_calibrate_898696_30.stderr parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/de91f86e-33db-4d6d-aa34-c0829536fa8e_characterizeImage_898696_30.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/de91f86e-33db-4d6d-aa34-c0829536fa8e_characterizeImage_898696_30.stderr parsl.dataflow.dflow INFO: Task 884 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acba211c60af95f87cd6741715c11d03 with result from task 884 parsl.dataflow.dflow DEBUG: Task 884 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/5a2103a6-f3f4-4829-bdff-b05c0288a4d3_calibrate_898798_22.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/5a2103a6-f3f4-4829-bdff-b05c0288a4d3_calibrate_898798_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:41 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8963f8b5afe42643d5b20b3f74fbcf0f with result from task 479 parsl.dataflow.dflow DEBUG: Task 479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/479fd069-8081-407e-9a79-44a06c69b06f_calibrate_898798_28.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/479fd069-8081-407e-9a79-44a06c69b06f_calibrate_898798_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:41 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:42 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:42 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:42 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:42 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71e9480734f0caf45603fcd9340e5292 with result from task 602 parsl.dataflow.dflow DEBUG: Task 602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/ade8159f-4e3e-4d16-b014-44cff63c7b89_calibrate_898800_23.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/ade8159f-4e3e-4d16-b014-44cff63c7b89_calibrate_898800_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:43 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:43 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:43 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:44 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:44 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c90f9fc95df0fbefeb3a1e1ff041311 with result from task 671 parsl.dataflow.dflow DEBUG: Task 671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/b659ea7e-a4f3-460c-beaf-54412afb9cef_calibrate_898799_1.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/b659ea7e-a4f3-460c-beaf-54412afb9cef_calibrate_898799_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:44 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:45 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:45 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:45 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:46 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:46 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 923 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3da1c062b14f3a9eeabe4704c476f13 with result from task 923 parsl.dataflow.dflow DEBUG: Task 923 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/0404a286-2598-424d-b0fe-943829e11c00_calibrate_898695_13.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/0404a286-2598-424d-b0fe-943829e11c00_calibrate_898695_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:46 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:47 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 799 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08ab8fb4c73f2340a2ce54baac0f28b7 with result from task 799 parsl.dataflow.dflow DEBUG: Task 799 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 800 has memoization hash 0a91882496ce78c2dd5b30a119dc7de4 parsl.dataflow.memoization INFO: Task 800 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 800 try 0 launched on executor multi with executor id 1108 parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/65eb2d3a-3f02-41fc-9660-eebfd213280e_calibrate_898800_19.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/65eb2d3a-3f02-41fc-9660-eebfd213280e_calibrate_898800_19.stderr parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/68ca4089-247f-463f-831a-a6c66b35297a_characterizeImage_898800_19.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/68ca4089-247f-463f-831a-a6c66b35297a_characterizeImage_898800_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:47 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:47 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:48 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3524b8329f418a72513a29067bb3f64 with result from task 331 parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:48 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash 84511cb5a61470fb30d10f2c320009e2 parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 1109 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/943e5ce6-9014-4f6f-9019-30f7f327b881_calibrate_898696_12.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/943e5ce6-9014-4f6f-9019-30f7f327b881_calibrate_898696_12.stderr parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/34fbd89e-e3c2-4f7f-8434-95250e671236_characterizeImage_898696_12.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/34fbd89e-e3c2-4f7f-8434-95250e671236_characterizeImage_898696_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:48 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:49 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 1039 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 405f829704b21da5516836501765e574 with result from task 1039 parsl.dataflow.dflow DEBUG: Task 1039 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1040 has memoization hash ec2005e56086d13ad5abcd9327ff714f parsl.dataflow.memoization INFO: Task 1040 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c32ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1040 try 0 launched on executor multi with executor id 1110 parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/e38cd05e-5d3e-490c-a064-961267220e92_calibrate_898800_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/e38cd05e-5d3e-490c-a064-961267220e92_calibrate_898800_33.stderr parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/3d3b8cee-0e42-4830-bbbe-07571de9a323_characterizeImage_898800_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/3d3b8cee-0e42-4830-bbbe-07571de9a323_characterizeImage_898800_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:49 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:49 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3181f7eb15308993c8ce269c97aebbcf with result from task 545 parsl.dataflow.dflow DEBUG: Task 545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/8bce8e9a-1e2c-4a22-9eb5-ba58485bace3_calibrate_898799_48.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/8bce8e9a-1e2c-4a22-9eb5-ba58485bace3_calibrate_898799_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:49 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:50 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:50 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:50 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:51 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:51 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:51 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 1082 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a401c6c4baa9c5fd77b4a698de604828 with result from task 1082 parsl.dataflow.dflow DEBUG: Task 1082 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/e228f848-423f-4d58-aed4-04b5e94400b0_calibrate_898695_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/e228f848-423f-4d58-aed4-04b5e94400b0_calibrate_898695_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:52 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:52 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e676e47f6f53df88d53e08ae13ee1a69 with result from task 41 parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/51f75315-ba90-4346-9319-be973560e480_calibrate_898798_61.stdout parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/51f75315-ba90-4346-9319-be973560e480_calibrate_898798_61.stderr parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:52 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:52 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 15 parsl.dataflow.strategy DEBUG: Executor multi has 15 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 15 parsl.dataflow.strategy DEBUG: Executor multi has 15 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.dflow INFO: Task 632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1ee3936b809e4027572c312171da12a with result from task 632 parsl.dataflow.dflow DEBUG: Task 632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/c7523ad6-8291-4407-95d4-2d97b846ff1b_calibrate_898696_49.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/c7523ad6-8291-4407-95d4-2d97b846ff1b_calibrate_898696_49.stderr 2024-03-13 09:50:58 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 09:50:58 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:58 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:58 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:58 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:59 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:59 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:59 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:50:59 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:00 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:00 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:00 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:00 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:00 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:01 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:01 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:01 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:01 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:02 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:02 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:02 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:02 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:02 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:03 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:03 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:03 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:03 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:04 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:04 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:04 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:04 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:04 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:05 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:05 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:05 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:05 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:06 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:06 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:06 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:06 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:07 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:07 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:07 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:07 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:07 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:08 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:08 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:08 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:08 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:09 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:09 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:09 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.dataflow.dflow INFO: Task 581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c74ba802dc820a19e170c65af1f4f9ed with result from task 581 parsl.dataflow.dflow DEBUG: Task 581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/d4727172-c61d-4cd9-bc37-29eefcb6061e_calibrate_898798_60.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898798/d4727172-c61d-4cd9-bc37-29eefcb6061e_calibrate_898798_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:09 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:09 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:10 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:10 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:10 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:10 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:10 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:11 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:11 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:11 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:11 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:11 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:12 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:12 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:12 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:12 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:12 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:12 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:13 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:13 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:13 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:13 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:13 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:14 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:14 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:14 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:14 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:14 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:15 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:15 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:15 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:15 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:15 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:16 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:16 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:16 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:16 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:16 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:17 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:17 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:17 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 13 parsl.dataflow.strategy DEBUG: Executor multi has 13 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 13 parsl.dataflow.strategy DEBUG: Executor multi has 13 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84511cb5a61470fb30d10f2c320009e2 with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/943e5ce6-9014-4f6f-9019-30f7f327b881_calibrate_898696_12.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/943e5ce6-9014-4f6f-9019-30f7f327b881_calibrate_898696_12.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 12 parsl.dataflow.strategy DEBUG: Executor multi has 12 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.dflow INFO: Task 661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5df32216f63aaa9d3ec01b0fe0b7cf79 with result from task 661 parsl.dataflow.dflow DEBUG: Task 661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 662 has memoization hash 9c960a3656e63692ff27392617d64f68 parsl.dataflow.memoization INFO: Task 662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c513f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 662 try 0 launched on executor multi with executor id 1111 parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/5f50ae5e-cff0-419c-8fde-03609c4a3305_calibrate_898694_16.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/5f50ae5e-cff0-419c-8fde-03609c4a3305_calibrate_898694_16.stderr parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/ff5d5a87-3752-446b-b651-b467c3e9ab77_characterizeImage_898694_16.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/ff5d5a87-3752-446b-b651-b467c3e9ab77_characterizeImage_898694_16.stderr parsl.dataflow.dflow INFO: Task 911 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24bcaaa7f86e276b47e47b8d207e2604 with result from task 911 parsl.dataflow.dflow DEBUG: Task 911 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/f208e14d-91f6-4f71-b325-767aeca7defc_calibrate_898800_4.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/f208e14d-91f6-4f71-b325-767aeca7defc_calibrate_898800_4.stderr parsl.dataflow.dflow INFO: Task 494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 890357a22897dc698e14a56947154fd7 with result from task 494 parsl.dataflow.dflow DEBUG: Task 494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/ad4849d7-78ca-4b1e-98c3-72f53ff79d8f_calibrate_898696_30.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/ad4849d7-78ca-4b1e-98c3-72f53ff79d8f_calibrate_898696_30.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 10 parsl.dataflow.strategy DEBUG: Executor multi has 10 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 2024-03-13 09:51:32 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 09:51:32 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:32 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:32 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:33 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 800 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a91882496ce78c2dd5b30a119dc7de4 with result from task 800 parsl.dataflow.dflow DEBUG: Task 800 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/65eb2d3a-3f02-41fc-9660-eebfd213280e_calibrate_898800_19.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/65eb2d3a-3f02-41fc-9660-eebfd213280e_calibrate_898800_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:33 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:33 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:33 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:33 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:33 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:33 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:34 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:34 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:34 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:34 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:34 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:34 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:34 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:35 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:35 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:35 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:35 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:35 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 1040 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec2005e56086d13ad5abcd9327ff714f with result from task 1040 parsl.dataflow.dflow DEBUG: Task 1040 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/e38cd05e-5d3e-490c-a064-961267220e92_calibrate_898800_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/e38cd05e-5d3e-490c-a064-961267220e92_calibrate_898800_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:35 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:35 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:35 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:36 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:36 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:36 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:36 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:36 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:36 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:36 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:37 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:37 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:37 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:37 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:37 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:37 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:37 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:37 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:38 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:38 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:38 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:38 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:38 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:38 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:38 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:39 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:39 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:39 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:39 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:39 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:39 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:39 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:39 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:40 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:40 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:40 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:40 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:40 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:40 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:40 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:41 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:41 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:41 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:41 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:41 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:41 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:41 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:42 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:42 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:42 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:42 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:42 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:42 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:42 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:43 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:43 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:43 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:43 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:43 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:43 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:43 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:43 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:44 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:44 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:44 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:44 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:44 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:44 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:44 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:51:45 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.dflow INFO: Task 662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c960a3656e63692ff27392617d64f68 with result from task 662 parsl.dataflow.dflow DEBUG: Task 662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/5f50ae5e-cff0-419c-8fde-03609c4a3305_calibrate_898694_16.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/5f50ae5e-cff0-419c-8fde-03609c4a3305_calibrate_898694_16.stderr parsl.dataflow.dflow INFO: Task 943 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ca89422d022310a130a59bef185ce80 with result from task 943 parsl.dataflow.dflow DEBUG: Task 943 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 944 has memoization hash 4493442c206d93cfd3152b2cd4537195 parsl.dataflow.memoization INFO: Task 944 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2922a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 944 try 0 launched on executor multi with executor id 1112 parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/f80ea3be-063d-48b5-a2ff-37f0fac8b405_calibrate_898800_61.stdout parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/f80ea3be-063d-48b5-a2ff-37f0fac8b405_calibrate_898800_61.stderr parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/c9844dfb-2978-4e5e-9da9-d236e1419f51_characterizeImage_898800_61.stdout parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898800/c9844dfb-2978-4e5e-9da9-d236e1419f51_characterizeImage_898800_61.stderr 2024-03-13 09:52:05 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 09:52:05 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:05 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:05 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:05 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:05 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:05 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:06 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:06 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:06 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:06 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:06 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:06 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:06 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:07 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:07 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:07 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:07 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:07 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:07 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:07 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:07 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:08 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:08 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:08 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:08 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:08 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:08 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:08 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:08 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:09 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:09 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:09 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:09 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:09 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:09 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:09 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:10 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:10 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:10 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:10 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:10 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:10 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:10 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:10 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:11 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:11 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:11 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:11 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:11 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:11 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:11 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:12 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:12 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:12 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:12 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:12 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:12 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:12 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:13 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:13 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:13 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:13 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:13 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:13 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:13 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:13 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:14 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:14 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:14 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:14 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:14 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:14 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:14 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:15 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:15 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:15 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:15 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:15 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:15 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:15 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:16 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:16 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:16 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:16 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:16 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:16 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:16 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:16 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:17 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:17 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:17 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:17 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 7 parsl.dataflow.strategy DEBUG: Executor multi has 7 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 7 parsl.dataflow.strategy DEBUG: Executor multi has 7 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 7 parsl.dataflow.strategy DEBUG: Executor multi has 7 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 7 parsl.dataflow.strategy DEBUG: Executor multi has 7 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 7 parsl.dataflow.strategy DEBUG: Executor multi has 7 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 2024-03-13 09:52:37 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 09:52:37 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:37 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:38 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:38 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:38 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:38 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:38 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:38 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:38 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:39 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:39 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:39 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:39 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:39 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:39 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:39 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:39 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:40 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:40 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:40 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:40 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:40 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:40 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:40 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:41 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:41 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:41 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:41 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:41 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:41 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:41 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:41 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:42 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:42 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:42 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:42 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:42 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:42 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:42 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:43 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:43 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:43 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:43 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:43 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:43 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:43 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:43 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:44 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:44 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:44 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:44 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:44 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:44 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:44 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:44 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:45 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:45 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:45 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:45 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:45 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:45 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:46 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:46 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:46 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:46 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:46 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:46 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:46 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:46 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:47 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:47 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:47 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:47 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:47 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:47 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:47 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:48 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:48 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:48 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:48 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:48 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:48 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:48 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:49 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:49 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:49 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:49 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:49 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:49 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:49 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:50 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:52:50 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 7 parsl.dataflow.strategy DEBUG: Executor multi has 7 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 7 parsl.dataflow.strategy DEBUG: Executor multi has 7 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.dflow INFO: Task 944 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4493442c206d93cfd3152b2cd4537195 with result from task 944 parsl.dataflow.dflow DEBUG: Task 944 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/f80ea3be-063d-48b5-a2ff-37f0fac8b405_calibrate_898800_61.stdout parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898800/f80ea3be-063d-48b5-a2ff-37f0fac8b405_calibrate_898800_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 2024-03-13 09:53:10 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 09:53:10 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:10 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:10 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:10 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:11 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:11 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:11 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:11 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:11 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:11 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:11 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:11 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:12 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:12 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:12 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:12 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:12 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:12 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:12 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:13 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:13 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:13 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:13 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:13 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:13 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:13 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:13 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:14 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:14 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:14 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:14 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:14 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:14 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:14 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:15 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:15 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:15 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:15 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:15 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:15 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:15 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:15 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:16 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:16 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:16 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:16 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:16 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:16 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:16 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:17 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:17 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:17 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:17 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:17 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:17 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:17 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:18 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:18 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:18 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:18 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:18 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:18 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:18 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:19 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:19 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:19 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:19 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:19 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:19 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:19 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:19 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:20 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:20 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:20 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:20 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:20 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:21 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:21 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:21 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:21 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:21 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:21 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:21 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:22 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:22 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:22 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:22 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:22 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:22 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:22 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:22 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:23 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.dflow INFO: Task 991 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c2a59281824f8a0c10f9afa7a9bd34d with result from task 991 parsl.dataflow.dflow DEBUG: Task 991 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 992 has memoization hash 63b2b4763af36368ea2713810f71e764 parsl.dataflow.memoization INFO: Task 992 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 992 try 0 launched on executor multi with executor id 1113 parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/a11f4193-a677-4128-b3d1-6c37c35f9d70_calibrate_898694_61.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/a11f4193-a677-4128-b3d1-6c37c35f9d70_calibrate_898694_61.stderr parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/bd18feba-85c1-4636-974c-48971fbe0f4c_characterizeImage_898694_61.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898694/bd18feba-85c1-4636-974c-48971fbe0f4c_characterizeImage_898694_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c276a4ba8598a2aa837d56eb7c7e98d3 with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash b32b6023f393ad7855a113323e977423 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d62ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 1114 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/bf9fa021-378b-4e5e-b634-d2b98ff17e27_calibrate_898696_28.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/bf9fa021-378b-4e5e-b634-d2b98ff17e27_calibrate_898696_28.stderr parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/cd5ab4bd-0725-4481-b89c-aa2b79a1fcfe_characterizeImage_898696_28.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/cd5ab4bd-0725-4481-b89c-aa2b79a1fcfe_characterizeImage_898696_28.stderr 2024-03-13 09:53:43 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 09:53:43 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:43 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:43 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:43 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:44 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:44 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:44 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:44 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:44 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:44 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:45 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:45 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:45 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:45 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:45 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:46 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:46 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:46 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:46 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:46 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:47 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:47 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:47 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:47 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:47 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:48 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:48 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:48 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:48 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:48 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:49 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:49 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:49 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:49 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:49 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:49 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:50 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:50 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:50 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:50 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:51 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:51 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:51 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:51 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:51 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:52 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:52 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:52 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:52 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:52 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:52 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:53 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:53 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:53 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:53 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:53 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:54 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:54 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:54 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:54 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:54 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:55 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:55 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:55 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:55 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:55 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:56 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:56 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:56 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:56 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:56 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:57 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:57 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:57 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:57 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:57 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:58 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:58 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:58 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:58 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:58 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:59 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:59 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:59 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:59 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:59 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:53:59 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:00 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:00 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:00 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:00 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:00 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.dflow INFO: Task 808 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f67f93d412a6eb955eaa2f10c1710b4a with result from task 808 parsl.dataflow.dflow DEBUG: Task 808 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 809 has memoization hash f5361f359844222310e742e00fd5b183 parsl.dataflow.memoization INFO: Task 809 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3f0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 809 try 0 launched on executor multi with executor id 1115 parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/b6b2fea8-8f20-450f-ba7b-97587fee8e0a_calibrate_898799_55.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/b6b2fea8-8f20-450f-ba7b-97587fee8e0a_calibrate_898799_55.stderr parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/03215a29-38e3-4542-b499-b8d432f0d075_characterizeImage_898799_55.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/03215a29-38e3-4542-b499-b8d432f0d075_characterizeImage_898799_55.stderr parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b32b6023f393ad7855a113323e977423 with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/bf9fa021-378b-4e5e-b634-d2b98ff17e27_calibrate_898696_28.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/bf9fa021-378b-4e5e-b634-d2b98ff17e27_calibrate_898696_28.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 992 try 0 failed parsl.dataflow.dflow INFO: Task 992 marked for retry parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/a11f4193-a677-4128-b3d1-6c37c35f9d70_calibrate_898694_61.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/a11f4193-a677-4128-b3d1-6c37c35f9d70_calibrate_898694_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 992 has memoization hash 63b2b4763af36368ea2713810f71e764 parsl.dataflow.memoization INFO: Task 992 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c2d9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 992 try 1 launched on executor multi with executor id 1116 parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/a11f4193-a677-4128-b3d1-6c37c35f9d70_calibrate_898694_61.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/a11f4193-a677-4128-b3d1-6c37c35f9d70_calibrate_898694_61.stderr 2024-03-13 09:54:16 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 09:54:16 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:16 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:16 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:16 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:16 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:16 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:17 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:17 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:17 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:17 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:17 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:17 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:17 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:17 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:18 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:18 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:18 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:18 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:18 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:18 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:19 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:19 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:19 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:19 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:19 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:19 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:19 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:19 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:20 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:20 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:20 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:20 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:20 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:21 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:21 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:21 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:21 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:21 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:21 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:21 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:22 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:22 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:22 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:22 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:22 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:22 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:22 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:23 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:23 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:23 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:23 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:23 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:23 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:23 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:23 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:24 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:24 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:24 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:24 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:24 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:24 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:24 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:25 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:25 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:25 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:25 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:25 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:25 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:25 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:26 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:26 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:26 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:26 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:26 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:26 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:26 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:27 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:27 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:27 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:27 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:27 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:27 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:27 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:27 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:28 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:28 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:28 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:28 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:28 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:28 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:28 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:29 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.dflow INFO: Task 790 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2670685054ccc60f57cb296510a66110 with result from task 790 parsl.dataflow.dflow DEBUG: Task 790 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 791 has memoization hash ed0b0c29ec7474591fa23cae1fb45ba4 parsl.dataflow.memoization INFO: Task 791 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c3bb880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 791 try 0 launched on executor multi with executor id 1117 parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/e480e6cf-a62a-4abc-aad1-9f584d3e79ea_calibrate_898799_61.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/e480e6cf-a62a-4abc-aad1-9f584d3e79ea_calibrate_898799_61.stderr parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/92ff6e5e-916a-42f8-8251-f0cd6370dffc_characterizeImage_898799_61.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898799/92ff6e5e-916a-42f8-8251-f0cd6370dffc_characterizeImage_898799_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.dflow INFO: Task 809 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5361f359844222310e742e00fd5b183 with result from task 809 parsl.dataflow.dflow DEBUG: Task 809 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/b6b2fea8-8f20-450f-ba7b-97587fee8e0a_calibrate_898799_55.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/b6b2fea8-8f20-450f-ba7b-97587fee8e0a_calibrate_898799_55.stderr 2024-03-13 09:54:49 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 09:54:49 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:49 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:49 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:49 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 992 try 1 failed parsl.dataflow.dflow ERROR: Task 992 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 63b2b4763af36368ea2713810f71e764 with result from task 992 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/a11f4193-a677-4128-b3d1-6c37c35f9d70_calibrate_898694_61.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898694/a11f4193-a677-4128-b3d1-6c37c35f9d70_calibrate_898694_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:49 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:49 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:50 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:50 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:50 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:50 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:50 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:50 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:51 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:51 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:51 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:51 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:51 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:51 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:51 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:52 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:52 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:52 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:52 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:52 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:52 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:52 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:52 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:53 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:53 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:53 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:53 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:53 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:53 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:53 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:54 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:54 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:54 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:54 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:54 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:54 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:54 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:54 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:55 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:55 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:55 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:55 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:55 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:55 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:55 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:56 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:56 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:56 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:56 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:56 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:56 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:56 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:56 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:57 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:57 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:57 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:57 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:57 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:57 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:57 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:58 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:58 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:58 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:58 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:58 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:58 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:58 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:58 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:59 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:59 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:59 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:59 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:59 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:59 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:59 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:54:59 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:00 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:00 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:00 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:00 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:00 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:00 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:00 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:01 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:01 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:01 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:01 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:01 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 2024-03-13 09:55:21 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 09:55:21 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:21 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:22 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:22 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:22 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:22 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:22 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:22 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:22 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:23 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:23 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:23 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:23 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:23 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:23 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:23 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:23 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:24 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:24 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:24 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:24 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:24 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:24 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:24 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:25 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:25 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:25 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:25 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:25 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.dataflow.dflow INFO: Task 791 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed0b0c29ec7474591fa23cae1fb45ba4 with result from task 791 parsl.dataflow.dflow DEBUG: Task 791 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/e480e6cf-a62a-4abc-aad1-9f584d3e79ea_calibrate_898799_61.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898799/e480e6cf-a62a-4abc-aad1-9f584d3e79ea_calibrate_898799_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:25 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:25 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:26 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:26 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:26 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:26 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:26 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:26 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:26 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:27 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:27 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:27 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:27 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:27 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:27 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:27 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:27 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:28 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:28 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:28 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:28 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:28 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:28 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:28 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:29 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:29 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:29 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:29 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:29 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:29 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:29 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:30 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:30 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:30 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:30 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:30 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:30 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:30 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:30 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:31 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:31 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:31 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:31 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:31 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:31 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:31 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:32 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:32 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:32 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:32 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:32 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:32 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:32 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:33 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:33 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:33 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:33 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:33 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:33 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:33 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:33 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:34 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:34 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 2024-03-13 09:55:54 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 09:55:54 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:54 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:54 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:54 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:55 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:55 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:55 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:55 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:55 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:55 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:55 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:56 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:56 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:56 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:56 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:56 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:56 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:56 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:57 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:57 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:57 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:57 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:57 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:57 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:57 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:58 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:58 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:58 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:58 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:58 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:58 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:58 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:58 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:59 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:59 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:59 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:59 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:59 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:59 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:55:59 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:00 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:00 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:00 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:00 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:00 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:00 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:00 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:01 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:01 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:01 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:01 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:01 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:01 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:01 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:02 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:02 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:02 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:02 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:02 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:02 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:03 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:03 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:03 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:03 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:03 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:03 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:03 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:04 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:04 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:04 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:04 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:04 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:04 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:04 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:05 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:05 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:05 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:05 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:05 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:05 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:05 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:06 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:06 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:06 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:06 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:06 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:06 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:06 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:07 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:07 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:07 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:07 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 2024-03-13 09:56:27 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 09:56:27 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:27 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:27 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:28 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:28 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:28 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:28 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:28 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:28 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:28 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:29 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:29 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:29 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:29 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:29 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:29 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:29 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:30 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:30 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:30 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:30 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:30 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:30 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:30 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:31 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:31 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:31 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:31 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:31 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:31 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:31 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:32 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:32 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:32 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:32 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:32 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:32 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:32 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:33 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:33 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:33 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:33 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:33 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:33 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:33 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:34 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:34 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:34 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:34 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:34 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:34 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:34 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:35 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:35 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:35 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:35 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:35 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:35 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:36 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:36 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:36 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:36 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:36 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:36 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:37 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:37 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:37 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:37 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:37 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:38 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:38 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:38 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:38 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:38 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:38 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:38 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:39 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:39 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:39 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:39 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:39 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:39 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:40 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:40 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:40 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:40 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:40 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:40 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:41 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:41 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:41 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:56:41 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 2024-03-13 09:57:01 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 09:57:01 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:01 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:01 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:02 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:02 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:02 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:02 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:02 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:02 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:02 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:03 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:03 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:03 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:03 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:03 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:03 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:03 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:04 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:04 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:04 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:04 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:04 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:04 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:04 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:05 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:05 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:05 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:05 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:05 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:05 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:05 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:06 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:06 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:06 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:06 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:06 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:06 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:07 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:07 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:07 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:07 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:07 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:07 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:07 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:08 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:08 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:08 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:08 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:08 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:08 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:08 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:09 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:09 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:09 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:09 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:09 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:09 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:09 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:10 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:10 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:10 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:10 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:10 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:10 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:10 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:11 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:11 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:11 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:11 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:11 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:11 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:11 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:12 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:12 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:12 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:12 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:12 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:12 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:12 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:13 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:13 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:13 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:13 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:13 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:13 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:13 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:14 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:14 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:14 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:14 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:14 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:14 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 2024-03-13 09:57:34 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 09:57:35 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:35 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:35 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:35 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:35 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:35 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:35 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:36 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:36 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:36 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:36 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:36 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:36 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:37 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:37 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:37 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:37 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:37 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:37 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:37 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:38 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:38 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:38 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:38 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:38 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:38 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:38 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:39 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:39 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:39 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:39 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:39 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:39 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:39 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:40 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:40 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:40 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:40 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:40 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:40 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:40 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:41 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:41 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:41 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:41 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:41 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:41 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:42 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:42 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:42 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:42 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:42 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:42 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:42 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:43 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:43 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:43 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:43 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:43 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:43 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:43 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:44 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:44 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:44 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:44 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:44 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:44 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:44 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:45 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:45 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:45 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:45 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:45 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:45 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:46 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:46 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:46 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:46 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:46 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:46 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:46 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:47 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:47 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:47 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:47 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:47 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:47 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:47 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:48 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:48 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:48 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:57:48 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12a2b88c13c94b84298bd5afe6890e97 with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 555c050f7e475ea958593a27b9a51fa8 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7d5ccc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 1118 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/90d2e8ca-1e2d-426c-b900-34065851121a_calibrate_898695_55.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/90d2e8ca-1e2d-426c-b900-34065851121a_calibrate_898695_55.stderr parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/97c76c2d-7941-48fd-b97e-55845ce70722_characterizeImage_898695_55.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898695/97c76c2d-7941-48fd-b97e-55845ce70722_characterizeImage_898695_55.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 2024-03-13 09:58:08 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 09:58:08 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:08 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:09 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:09 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:09 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:09 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:09 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:09 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:09 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:10 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:10 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:10 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:10 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:10 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:10 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:10 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:11 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:11 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:11 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:11 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:11 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:11 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:11 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:12 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:12 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:12 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:12 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:12 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:12 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:12 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:13 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:13 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:13 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:13 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:13 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:13 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:13 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:14 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:14 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:14 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:14 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:14 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:14 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:15 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:15 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:15 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:15 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:15 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:15 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:15 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:16 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:16 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:16 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:16 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:16 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:16 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:16 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:17 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:17 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:17 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:17 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:17 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:17 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:18 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:18 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:18 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:18 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:18 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:18 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:19 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:19 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:19 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:19 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:19 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:19 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:19 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:20 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:20 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:20 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:20 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:20 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:20 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:21 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:21 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:21 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:21 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:21 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:21 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:21 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:22 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:22 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:22 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 555c050f7e475ea958593a27b9a51fa8 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/90d2e8ca-1e2d-426c-b900-34065851121a_calibrate_898695_55.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898695/90d2e8ca-1e2d-426c-b900-34065851121a_calibrate_898695_55.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 2024-03-13 09:58:42 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 09:58:42 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:42 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:42 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:43 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:43 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:43 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:43 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:43 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:43 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:43 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:44 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:44 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:44 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:44 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:44 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:44 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:44 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:45 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:45 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:45 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:45 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:45 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:45 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:45 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:46 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:46 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:46 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:46 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:46 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:46 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:46 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:47 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:47 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:47 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:47 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:47 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:47 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:47 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:48 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:48 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:48 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:48 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:48 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:48 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:48 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:49 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:49 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:49 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:49 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:49 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:49 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:49 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:50 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:50 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:50 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:50 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:50 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:50 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:50 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:51 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:51 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:51 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:51 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:51 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:51 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:51 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:52 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:52 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:52 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:52 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:52 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:52 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:53 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:53 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:53 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:53 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:53 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:53 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:53 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:54 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:54 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:54 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:54 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:54 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:54 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:54 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:55 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:55 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:55 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:55 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:55 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:58:55 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 2024-03-13 09:59:16 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 09:59:16 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:16 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:16 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:16 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:16 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:16 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:16 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:17 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:17 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:17 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:17 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:17 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:17 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:17 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:18 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:18 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:18 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:18 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:18 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:19 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:19 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:19 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:19 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:19 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:19 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:20 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:20 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:20 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:20 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:20 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:20 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:20 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:21 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:21 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:21 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:21 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:21 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:21 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:21 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:22 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:22 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:22 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:22 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:22 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:22 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:23 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:23 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:23 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:23 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:23 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:23 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:23 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:24 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:24 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:24 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:24 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:24 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:24 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:25 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:25 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:25 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:25 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:25 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:25 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:25 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:26 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:26 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:26 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:26 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:26 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:26 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:26 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:27 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:27 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:27 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:27 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:27 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:27 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:27 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:28 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:28 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:28 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:28 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:28 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:28 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:29 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:29 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:29 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:29 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:29 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:29 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:29 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 2024-03-13 09:59:50 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 09:59:50 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:50 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:50 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:50 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:50 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:50 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:50 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:51 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:51 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:51 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:51 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:51 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:51 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:51 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:52 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:52 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:52 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:52 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:52 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:52 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:53 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:53 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:53 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:53 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:53 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:53 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:54 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:54 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:54 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:54 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:54 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:54 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:54 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:55 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:55 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:55 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:55 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:55 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:55 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:55 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:56 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:56 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:56 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:56 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:56 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:56 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:56 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:57 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:57 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:57 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:57 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:57 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:57 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:57 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:58 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:58 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:58 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:58 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:58 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:58 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:59 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:59 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:59 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:59 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:59 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:59 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 09:59:59 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:00 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:00 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:00 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:00 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:00 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:00 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:00 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:01 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:01 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:01 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:01 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:01 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:01 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:01 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:02 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:02 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:02 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:02 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:02 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:02 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:02 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:03 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:03 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:03 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:03 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.dflow INFO: Task 1060 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e961373062c64b635363c50c3a58406 with result from task 1060 parsl.dataflow.dflow DEBUG: Task 1060 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1061 has memoization hash 47f5d6b34e00a02134633ef26757607d parsl.dataflow.memoization INFO: Task 1061 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151b7c328680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1061 try 0 launched on executor multi with executor id 1119 parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/5d2ef2e1-8cb4-4f97-a76d-40d07ea65ff6_calibrate_898696_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/5d2ef2e1-8cb4-4f97-a76d-40d07ea65ff6_calibrate_898696_61.stderr parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/96906a5e-0eca-4503-963d-cc645fe2cbf8_characterizeImage_898696_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/characterizeImage/898696/96906a5e-0eca-4503-963d-cc645fe2cbf8_characterizeImage_898696_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 2024-03-13 10:00:23 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 10:00:23 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:23 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:24 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:24 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:24 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:24 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:24 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:24 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:24 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:25 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:25 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:25 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:25 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:25 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:25 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:25 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:26 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:26 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:26 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:26 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:26 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:26 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:26 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:27 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:27 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:27 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:27 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:27 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:27 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:27 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010046,17010047,17010048,17010049,17010050,17010051,17010052,17010053,17010054,17010055,17010056,17010057,17010058,17010059,17010060,17010061,17010062,17010063,17010064,17010065,17010066,17010067,17010069,17010072,17010074,17010075,17010076,17010077,17010078,17010080' parsl.providers.slurm.slurm DEBUG: squeue returned 17010078 R 17010080 R 17010076 R 17010077 R 17010075 R 17010072 R 17010074 R 17010065 R 17010066 R 17010067 R 17010069 R 17010061 R 17010062 R 17010063 R 17010064 R 17010056 R 17010057 R 17010058 R 17010059 R 17010060 R 17010046 R 17010047 R 17010048 R 17010049 R 17010050 R 17010051 R 17010052 R 17010053 R 17010054 R 17010055 R parsl.providers.slurm.slurm DEBUG: Updating job 17010078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010063 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010055 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:28 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:28 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:28 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:28 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:28 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010081,17010082,17010084,17010086,17010087' parsl.providers.slurm.slurm DEBUG: squeue returned 17010087 R 17010084 R 17010086 R 17010081 R 17010082 R parsl.providers.slurm.slurm DEBUG: Updating job 17010087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010082 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:28 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:29 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:29 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:29 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:29 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:29 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:29 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:29 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:30 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:30 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:30 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:30 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:30 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:30 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:30 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:31 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:31 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:31 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:31 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:31 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010088,17010089,17010091,17010092,17010093,17010094,17010095,17010096,17010097,17010098,17010099,17010100,17010101,17010102,17010104,17010105,17010106,17010107,17010109,17010110' parsl.providers.slurm.slurm DEBUG: squeue returned 17010110 R 17010106 R 17010107 R 17010109 R 17010104 R 17010105 R 17010100 R 17010101 R 17010102 R 17010099 R 17010097 R 17010098 R 17010096 R 17010094 R 17010095 R 17010092 R 17010093 R 17010088 R 17010089 R 17010091 R parsl.providers.slurm.slurm DEBUG: Updating job 17010110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010091 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:31 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:31 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:32 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:32 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:32 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:32 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:32 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:32 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:33 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:33 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:33 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:33 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:33 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:33 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:34 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:34 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:34 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:34 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:34 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:34 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:34 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:35 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:35 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:35 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:35 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:35 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:35 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:35 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:36 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:36 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:36 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:36 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:36 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:36 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:37 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:37 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING 2024-03-13 10:00:37 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17010111,17010112,17010113,17010114,17010115,17010116,17010117,17010118,17010119,17010120,17010122,17010123,17010124,17010125,17010126,17010127,17010128,17010129,17010130,17010131,17010132,17010133,17010134,17010135,17010136,17010137,17010139,17010140,17010141,17010142,17010143,17010144,17010145,17010146,17010148,17010149,17010150' parsl.providers.slurm.slurm DEBUG: squeue returned 17010128 R 17010129 R 17010130 R 17010131 R 17010132 R 17010133 R 17010134 R 17010135 R 17010136 R 17010137 R 17010139 R 17010140 R 17010141 R 17010142 R 17010143 R 17010144 R 17010145 R 17010146 R 17010148 R 17010149 R 17010150 R 17010127 R 17010111 R 17010112 R 17010113 R 17010114 R 17010115 R 17010116 R 17010117 R 17010118 R 17010119 R 17010120 R 17010122 R 17010123 R 17010124 R 17010125 R 17010126 R parsl.providers.slurm.slurm DEBUG: Updating job 17010128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17010126 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23207851629712 parsl.dataflow.dflow INFO: Task 1061 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47f5d6b34e00a02134633ef26757607d with result from task 1061 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow DEBUG: Task 1061 checkpointed parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 1101 parsl.dataflow.dflow INFO: Tasks in state States.failed: 9 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 6 parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/5d2ef2e1-8cb4-4f97-a76d-40d07ea65ff6_calibrate_898696_61.stdout parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/submit/DEEP/20190927/B1a/science#step1/20240313T094419Z/logs/calibrate/898696/5d2ef2e1-8cb4-4f97-a76d-40d07ea65ff6_calibrate_898696_61.stderr parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=93, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 93 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ea1eb8e003a4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ea1eb8e003a4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4a8bbc2bfa67 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4a8bbc2bfa67 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7dc04efc07e2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7dc04efc07e2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 60bd1c6e6317 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 60bd1c6e6317 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f47ec62fe2f8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f47ec62fe2f8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 417603e8c6c7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 417603e8c6c7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b220a4a19346 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b220a4a19346 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ddd035540400 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ddd035540400 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a42fe3501645 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a42fe3501645 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3c0570c2a23d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3c0570c2a23d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a63fb39ad072 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a63fb39ad072 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f391ee9e3c23 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f391ee9e3c23 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f05506b7fe41 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f05506b7fe41 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 60b2b4a3975f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 60b2b4a3975f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5f822bc885b5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5f822bc885b5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bad847665360 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bad847665360 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0dd3e2110d87 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0dd3e2110d87 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dd036bd4219e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dd036bd4219e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 41cf3aa9b4aa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 41cf3aa9b4aa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e9b0a83c6bd6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e9b0a83c6bd6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a8ba3bcc77d9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a8ba3bcc77d9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c9d29830056d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c9d29830056d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6b2e0b3c43ab parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6b2e0b3c43ab parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a8a1d98b7529 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a8a1d98b7529 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 536ade796ad1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 536ade796ad1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7f7cb5b47186 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7f7cb5b47186 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4230179780bc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4230179780bc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 639cf716e125 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 639cf716e125 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 315f844bd6e5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 315f844bd6e5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a92ea9bf4a0d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a92ea9bf4a0d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e40060086f1b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e40060086f1b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8878a185c5ba parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8878a185c5ba parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: af1807a3b307 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: af1807a3b307 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2bdf5b5cb755 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2bdf5b5cb755 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 466bec477f67 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 466bec477f67 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2ce4c9c5dc8d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2ce4c9c5dc8d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8ccf9affd0e7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8ccf9affd0e7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7c89c8606def parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7c89c8606def parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 884f96934043 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 884f96934043 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 18e750bd9a6a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 18e750bd9a6a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 398a77dbd3d4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 398a77dbd3d4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bad6cc1f202a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bad6cc1f202a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a02514786363 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a02514786363 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: faddc703b36f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: faddc703b36f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a03130a95099 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a03130a95099 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e07f08a1b3f3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e07f08a1b3f3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 91d6c1a0f54f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 91d6c1a0f54f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cd08d780d77e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cd08d780d77e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f77553f7b6a9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f77553f7b6a9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dd21d0e3cda9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dd21d0e3cda9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f2d44fda2fe6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f2d44fda2fe6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8d8d68c8c559 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8d8d68c8c559 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 04bfba19e198 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 04bfba19e198 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f3de56e50674 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f3de56e50674 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 525e91e1f080 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 525e91e1f080 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5c8828e1d3c3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5c8828e1d3c3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 913d99d907a4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 913d99d907a4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ccd6c6ce79af parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ccd6c6ce79af parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e521dd02bd4b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e521dd02bd4b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 01049be68e3d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 01049be68e3d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cf9f07fd4548 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cf9f07fd4548 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ae7cb1d4ecb7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ae7cb1d4ecb7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 93990e871342 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 93990e871342 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ac67c50f2c08 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ac67c50f2c08 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: afbbd05c1625 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: afbbd05c1625 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c9dc31b16f03 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c9dc31b16f03 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 49bb9704463c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 49bb9704463c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 11e42d287e3f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 11e42d287e3f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ce4f25413bad parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ce4f25413bad parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fd6c5d474903 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fd6c5d474903 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b6f541a8ed56 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b6f541a8ed56 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8d8fd7df2b6c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8d8fd7df2b6c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 43f60dd32f34 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 43f60dd32f34 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: abd64aea403f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: abd64aea403f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1e3afc79b51d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1e3afc79b51d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6fb648c946e5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6fb648c946e5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cf341c62ce9e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cf341c62ce9e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 369f460a8602 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 369f460a8602 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 14d139b3177f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 14d139b3177f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b3e43a80f44f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b3e43a80f44f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a0f748a6167c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a0f748a6167c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e31697bf96b5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e31697bf96b5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f601ebe78a03 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f601ebe78a03 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cd123763c81e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cd123763c81e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b6f00a8d4538 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b6f00a8d4538 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4370ca2dede1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4370ca2dede1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 051a240d0637 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 051a240d0637 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 91845d7bd322 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 91845d7bd322 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f6494db1c9fe parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f6494db1c9fe parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3adf5b93c86b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3adf5b93c86b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9091a8653853 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9091a8653853 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cf8f99966d95 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cf8f99966d95 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6afc6b9f800d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6afc6b9f800d 2024-03-13 10:00:46 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint 2024-03-13 10:00:46 proc_lsst.multi:201 [INFO] cancelling 17010119 on provider checkpoint proc_lsst.multi INFO: cancelling 17010119 on provider checkpoint 2024-03-13 10:00:46 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint 2024-03-13 10:00:46 proc_lsst.multi:201 [INFO] cancelling 17010150 on provider checkpoint proc_lsst.multi INFO: cancelling 17010150 on provider checkpoint 2024-03-13 10:00:46 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k 2024-03-13 10:00:46 proc_lsst.multi:201 [INFO] cancelling 17010086 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17010086 on provider escience_rtx6k 2024-03-13 10:00:46 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint 2024-03-13 10:00:46 proc_lsst.multi:201 [INFO] cancelling 17010118 on provider checkpoint proc_lsst.multi INFO: cancelling 17010118 on provider checkpoint 2024-03-13 10:00:46 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint 2024-03-13 10:00:46 proc_lsst.multi:201 [INFO] cancelling 17010148 on provider checkpoint proc_lsst.multi INFO: cancelling 17010148 on provider checkpoint 2024-03-13 10:00:46 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 2024-03-13 10:00:46 proc_lsst.multi:201 [INFO] cancelling 17010110 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010110 on provider escience_a40 2024-03-13 10:00:46 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint 2024-03-13 10:00:46 proc_lsst.multi:201 [INFO] cancelling 17010141 on provider checkpoint proc_lsst.multi INFO: cancelling 17010141 on provider checkpoint 2024-03-13 10:00:46 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint 2024-03-13 10:00:46 proc_lsst.multi:201 [INFO] cancelling 17010134 on provider checkpoint proc_lsst.multi INFO: cancelling 17010134 on provider checkpoint 2024-03-13 10:00:47 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 2024-03-13 10:00:47 proc_lsst.multi:201 [INFO] cancelling 17010106 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010106 on provider escience_a40 2024-03-13 10:00:47 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro 2024-03-13 10:00:47 proc_lsst.multi:201 [INFO] cancelling 17010053 on provider astro proc_lsst.multi INFO: cancelling 17010053 on provider astro 2024-03-13 10:00:47 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint 2024-03-13 10:00:47 proc_lsst.multi:201 [INFO] cancelling 17010139 on provider checkpoint proc_lsst.multi INFO: cancelling 17010139 on provider checkpoint 2024-03-13 10:00:47 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint 2024-03-13 10:00:47 proc_lsst.multi:201 [INFO] cancelling 17010112 on provider checkpoint proc_lsst.multi INFO: cancelling 17010112 on provider checkpoint 2024-03-13 10:00:47 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro 2024-03-13 10:00:47 proc_lsst.multi:201 [INFO] cancelling 17010066 on provider astro proc_lsst.multi INFO: cancelling 17010066 on provider astro 2024-03-13 10:00:47 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint 2024-03-13 10:00:47 proc_lsst.multi:201 [INFO] cancelling 17010120 on provider checkpoint proc_lsst.multi INFO: cancelling 17010120 on provider checkpoint 2024-03-13 10:00:47 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 2024-03-13 10:00:47 proc_lsst.multi:201 [INFO] cancelling 17010102 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010102 on provider escience_a40 2024-03-13 10:00:47 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro 2024-03-13 10:00:47 proc_lsst.multi:201 [INFO] cancelling 17010074 on provider astro proc_lsst.multi INFO: cancelling 17010074 on provider astro 2024-03-13 10:00:47 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 2024-03-13 10:00:47 proc_lsst.multi:201 [INFO] cancelling 17010107 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010107 on provider escience_a40 2024-03-13 10:00:48 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro 2024-03-13 10:00:48 proc_lsst.multi:201 [INFO] cancelling 17010064 on provider astro proc_lsst.multi INFO: cancelling 17010064 on provider astro 2024-03-13 10:00:48 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro 2024-03-13 10:00:48 proc_lsst.multi:201 [INFO] cancelling 17010058 on provider astro proc_lsst.multi INFO: cancelling 17010058 on provider astro 2024-03-13 10:00:48 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro 2024-03-13 10:00:48 proc_lsst.multi:201 [INFO] cancelling 17010060 on provider astro proc_lsst.multi INFO: cancelling 17010060 on provider astro 2024-03-13 10:00:48 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro 2024-03-13 10:00:48 proc_lsst.multi:201 [INFO] cancelling 17010075 on provider astro proc_lsst.multi INFO: cancelling 17010075 on provider astro 2024-03-13 10:00:48 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro 2024-03-13 10:00:48 proc_lsst.multi:201 [INFO] cancelling 17010055 on provider astro proc_lsst.multi INFO: cancelling 17010055 on provider astro 2024-03-13 10:00:48 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro 2024-03-13 10:00:48 proc_lsst.multi:201 [INFO] cancelling 17010063 on provider astro proc_lsst.multi INFO: cancelling 17010063 on provider astro 2024-03-13 10:00:48 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro 2024-03-13 10:00:48 proc_lsst.multi:201 [INFO] cancelling 17010049 on provider astro proc_lsst.multi INFO: cancelling 17010049 on provider astro 2024-03-13 10:00:48 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint 2024-03-13 10:00:48 proc_lsst.multi:201 [INFO] cancelling 17010136 on provider checkpoint proc_lsst.multi INFO: cancelling 17010136 on provider checkpoint 2024-03-13 10:00:48 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint 2024-03-13 10:00:48 proc_lsst.multi:201 [INFO] cancelling 17010132 on provider checkpoint proc_lsst.multi INFO: cancelling 17010132 on provider checkpoint 2024-03-13 10:00:49 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint 2024-03-13 10:00:49 proc_lsst.multi:201 [INFO] cancelling 17010145 on provider checkpoint proc_lsst.multi INFO: cancelling 17010145 on provider checkpoint 2024-03-13 10:00:49 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro 2024-03-13 10:00:49 proc_lsst.multi:201 [INFO] cancelling 17010072 on provider astro proc_lsst.multi INFO: cancelling 17010072 on provider astro 2024-03-13 10:00:49 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint 2024-03-13 10:00:49 proc_lsst.multi:201 [INFO] cancelling 17010133 on provider checkpoint proc_lsst.multi INFO: cancelling 17010133 on provider checkpoint 2024-03-13 10:00:49 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 2024-03-13 10:00:49 proc_lsst.multi:201 [INFO] cancelling 17010088 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010088 on provider escience_a40 2024-03-13 10:00:49 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint 2024-03-13 10:00:49 proc_lsst.multi:201 [INFO] cancelling 17010116 on provider checkpoint proc_lsst.multi INFO: cancelling 17010116 on provider checkpoint 2024-03-13 10:00:49 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k 2024-03-13 10:00:49 proc_lsst.multi:201 [INFO] cancelling 17010081 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17010081 on provider escience_rtx6k 2024-03-13 10:00:49 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro 2024-03-13 10:00:49 proc_lsst.multi:201 [INFO] cancelling 17010051 on provider astro proc_lsst.multi INFO: cancelling 17010051 on provider astro 2024-03-13 10:00:49 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint 2024-03-13 10:00:49 proc_lsst.multi:201 [INFO] cancelling 17010125 on provider checkpoint proc_lsst.multi INFO: cancelling 17010125 on provider checkpoint 2024-03-13 10:00:50 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k 2024-03-13 10:00:50 proc_lsst.multi:201 [INFO] cancelling 17010082 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17010082 on provider escience_rtx6k 2024-03-13 10:00:50 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint 2024-03-13 10:00:50 proc_lsst.multi:201 [INFO] cancelling 17010130 on provider checkpoint proc_lsst.multi INFO: cancelling 17010130 on provider checkpoint 2024-03-13 10:00:50 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint 2024-03-13 10:00:50 proc_lsst.multi:201 [INFO] cancelling 17010143 on provider checkpoint proc_lsst.multi INFO: cancelling 17010143 on provider checkpoint 2024-03-13 10:00:50 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro 2024-03-13 10:00:50 proc_lsst.multi:201 [INFO] cancelling 17010046 on provider astro proc_lsst.multi INFO: cancelling 17010046 on provider astro 2024-03-13 10:00:50 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro 2024-03-13 10:00:50 proc_lsst.multi:201 [INFO] cancelling 17010065 on provider astro proc_lsst.multi INFO: cancelling 17010065 on provider astro 2024-03-13 10:00:50 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint 2024-03-13 10:00:50 proc_lsst.multi:201 [INFO] cancelling 17010113 on provider checkpoint proc_lsst.multi INFO: cancelling 17010113 on provider checkpoint 2024-03-13 10:00:50 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 2024-03-13 10:00:50 proc_lsst.multi:201 [INFO] cancelling 17010093 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010093 on provider escience_a40 2024-03-13 10:00:50 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint 2024-03-13 10:00:50 proc_lsst.multi:201 [INFO] cancelling 17010140 on provider checkpoint proc_lsst.multi INFO: cancelling 17010140 on provider checkpoint 2024-03-13 10:00:50 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro 2024-03-13 10:00:50 proc_lsst.multi:201 [INFO] cancelling 17010059 on provider astro proc_lsst.multi INFO: cancelling 17010059 on provider astro 2024-03-13 10:00:51 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint 2024-03-13 10:00:51 proc_lsst.multi:201 [INFO] cancelling 17010124 on provider checkpoint proc_lsst.multi INFO: cancelling 17010124 on provider checkpoint 2024-03-13 10:00:51 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 2024-03-13 10:00:51 proc_lsst.multi:201 [INFO] cancelling 17010094 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010094 on provider escience_a40 2024-03-13 10:00:51 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint 2024-03-13 10:00:51 proc_lsst.multi:201 [INFO] cancelling 17010115 on provider checkpoint proc_lsst.multi INFO: cancelling 17010115 on provider checkpoint 2024-03-13 10:00:51 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint 2024-03-13 10:00:51 proc_lsst.multi:201 [INFO] cancelling 17010137 on provider checkpoint proc_lsst.multi INFO: cancelling 17010137 on provider checkpoint 2024-03-13 10:00:51 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 2024-03-13 10:00:51 proc_lsst.multi:201 [INFO] cancelling 17010109 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010109 on provider escience_a40 2024-03-13 10:00:51 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint 2024-03-13 10:00:51 proc_lsst.multi:201 [INFO] cancelling 17010122 on provider checkpoint proc_lsst.multi INFO: cancelling 17010122 on provider checkpoint 2024-03-13 10:00:51 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 2024-03-13 10:00:51 proc_lsst.multi:201 [INFO] cancelling 17010089 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010089 on provider escience_a40 2024-03-13 10:00:51 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint 2024-03-13 10:00:51 proc_lsst.multi:201 [INFO] cancelling 17010149 on provider checkpoint proc_lsst.multi INFO: cancelling 17010149 on provider checkpoint 2024-03-13 10:00:51 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint 2024-03-13 10:00:51 proc_lsst.multi:201 [INFO] cancelling 17010144 on provider checkpoint proc_lsst.multi INFO: cancelling 17010144 on provider checkpoint 2024-03-13 10:00:52 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint 2024-03-13 10:00:52 proc_lsst.multi:201 [INFO] cancelling 17010142 on provider checkpoint proc_lsst.multi INFO: cancelling 17010142 on provider checkpoint 2024-03-13 10:00:52 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint 2024-03-13 10:00:52 proc_lsst.multi:201 [INFO] cancelling 17010146 on provider checkpoint proc_lsst.multi INFO: cancelling 17010146 on provider checkpoint 2024-03-13 10:00:52 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro 2024-03-13 10:00:52 proc_lsst.multi:201 [INFO] cancelling 17010057 on provider astro proc_lsst.multi INFO: cancelling 17010057 on provider astro 2024-03-13 10:00:52 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro 2024-03-13 10:00:52 proc_lsst.multi:201 [INFO] cancelling 17010050 on provider astro proc_lsst.multi INFO: cancelling 17010050 on provider astro 2024-03-13 10:00:52 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 2024-03-13 10:00:52 proc_lsst.multi:201 [INFO] cancelling 17010096 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010096 on provider escience_a40 2024-03-13 10:00:52 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint 2024-03-13 10:00:52 proc_lsst.multi:201 [INFO] cancelling 17010127 on provider checkpoint proc_lsst.multi INFO: cancelling 17010127 on provider checkpoint 2024-03-13 10:00:52 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint 2024-03-13 10:00:52 proc_lsst.multi:201 [INFO] cancelling 17010117 on provider checkpoint proc_lsst.multi INFO: cancelling 17010117 on provider checkpoint 2024-03-13 10:00:52 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint 2024-03-13 10:00:52 proc_lsst.multi:201 [INFO] cancelling 17010111 on provider checkpoint proc_lsst.multi INFO: cancelling 17010111 on provider checkpoint 2024-03-13 10:00:52 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint 2024-03-13 10:00:52 proc_lsst.multi:201 [INFO] cancelling 17010131 on provider checkpoint proc_lsst.multi INFO: cancelling 17010131 on provider checkpoint 2024-03-13 10:00:53 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k 2024-03-13 10:00:53 proc_lsst.multi:201 [INFO] cancelling 17010084 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17010084 on provider escience_rtx6k 2024-03-13 10:00:53 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro 2024-03-13 10:00:53 proc_lsst.multi:201 [INFO] cancelling 17010048 on provider astro proc_lsst.multi INFO: cancelling 17010048 on provider astro 2024-03-13 10:00:53 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint 2024-03-13 10:00:53 proc_lsst.multi:201 [INFO] cancelling 17010128 on provider checkpoint proc_lsst.multi INFO: cancelling 17010128 on provider checkpoint 2024-03-13 10:00:53 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 2024-03-13 10:00:53 proc_lsst.multi:201 [INFO] cancelling 17010091 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010091 on provider escience_a40 2024-03-13 10:00:53 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint 2024-03-13 10:00:53 proc_lsst.multi:201 [INFO] cancelling 17010129 on provider checkpoint proc_lsst.multi INFO: cancelling 17010129 on provider checkpoint 2024-03-13 10:00:53 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro 2024-03-13 10:00:53 proc_lsst.multi:201 [INFO] cancelling 17010047 on provider astro proc_lsst.multi INFO: cancelling 17010047 on provider astro 2024-03-13 10:00:53 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint 2024-03-13 10:00:53 proc_lsst.multi:201 [INFO] cancelling 17010123 on provider checkpoint proc_lsst.multi INFO: cancelling 17010123 on provider checkpoint 2024-03-13 10:00:53 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 10:00:53 proc_lsst.multi:201 [INFO] cancelling 60730 on provider local proc_lsst.multi INFO: cancelling 60730 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 60730 2024-03-13 10:00:53 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro 2024-03-13 10:00:53 proc_lsst.multi:201 [INFO] cancelling 17010078 on provider astro proc_lsst.multi INFO: cancelling 17010078 on provider astro 2024-03-13 10:00:53 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro 2024-03-13 10:00:53 proc_lsst.multi:201 [INFO] cancelling 17010067 on provider astro proc_lsst.multi INFO: cancelling 17010067 on provider astro 2024-03-13 10:00:54 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro 2024-03-13 10:00:54 proc_lsst.multi:201 [INFO] cancelling 17010052 on provider astro proc_lsst.multi INFO: cancelling 17010052 on provider astro 2024-03-13 10:00:54 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint 2024-03-13 10:00:54 proc_lsst.multi:201 [INFO] cancelling 17010135 on provider checkpoint proc_lsst.multi INFO: cancelling 17010135 on provider checkpoint 2024-03-13 10:00:54 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint 2024-03-13 10:00:54 proc_lsst.multi:201 [INFO] cancelling 17010114 on provider checkpoint proc_lsst.multi INFO: cancelling 17010114 on provider checkpoint 2024-03-13 10:00:54 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 2024-03-13 10:00:54 proc_lsst.multi:201 [INFO] cancelling 17010092 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010092 on provider escience_a40 2024-03-13 10:00:54 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint 2024-03-13 10:00:54 proc_lsst.multi:201 [INFO] cancelling 17010126 on provider checkpoint proc_lsst.multi INFO: cancelling 17010126 on provider checkpoint 2024-03-13 10:00:54 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro 2024-03-13 10:00:54 proc_lsst.multi:201 [INFO] cancelling 17010062 on provider astro proc_lsst.multi INFO: cancelling 17010062 on provider astro 2024-03-13 10:00:54 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 2024-03-13 10:00:54 proc_lsst.multi:201 [INFO] cancelling 17010100 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010100 on provider escience_a40 2024-03-13 10:00:54 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 2024-03-13 10:00:54 proc_lsst.multi:201 [INFO] cancelling 17010104 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010104 on provider escience_a40 2024-03-13 10:00:54 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 2024-03-13 10:00:54 proc_lsst.multi:201 [INFO] cancelling 17010095 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010095 on provider escience_a40 2024-03-13 10:00:55 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro 2024-03-13 10:00:55 proc_lsst.multi:201 [INFO] cancelling 17010056 on provider astro proc_lsst.multi INFO: cancelling 17010056 on provider astro 2024-03-13 10:00:55 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 2024-03-13 10:00:55 proc_lsst.multi:201 [INFO] cancelling 17010105 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010105 on provider escience_a40 2024-03-13 10:00:55 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro 2024-03-13 10:00:55 proc_lsst.multi:201 [INFO] cancelling 17010061 on provider astro proc_lsst.multi INFO: cancelling 17010061 on provider astro 2024-03-13 10:00:55 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 2024-03-13 10:00:55 proc_lsst.multi:201 [INFO] cancelling 17010101 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010101 on provider escience_a40 2024-03-13 10:00:55 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro 2024-03-13 10:00:55 proc_lsst.multi:201 [INFO] cancelling 17010080 on provider astro proc_lsst.multi INFO: cancelling 17010080 on provider astro 2024-03-13 10:00:55 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 2024-03-13 10:00:55 proc_lsst.multi:201 [INFO] cancelling 17010098 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010098 on provider escience_a40 2024-03-13 10:00:55 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro 2024-03-13 10:00:55 proc_lsst.multi:201 [INFO] cancelling 17010077 on provider astro proc_lsst.multi INFO: cancelling 17010077 on provider astro 2024-03-13 10:00:55 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 2024-03-13 10:00:55 proc_lsst.multi:201 [INFO] cancelling 17010099 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010099 on provider escience_a40 2024-03-13 10:00:55 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro 2024-03-13 10:00:55 proc_lsst.multi:201 [INFO] cancelling 17010054 on provider astro proc_lsst.multi INFO: cancelling 17010054 on provider astro 2024-03-13 10:00:55 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 2024-03-13 10:00:55 proc_lsst.multi:201 [INFO] cancelling 17010097 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010097 on provider escience_a40 2024-03-13 10:00:56 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k 2024-03-13 10:00:56 proc_lsst.multi:201 [INFO] cancelling 17010087 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17010087 on provider escience_rtx6k 2024-03-13 10:00:56 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro 2024-03-13 10:00:56 proc_lsst.multi:201 [INFO] cancelling 17010076 on provider astro proc_lsst.multi INFO: cancelling 17010076 on provider astro 2024-03-13 10:00:56 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro 2024-03-13 10:00:56 proc_lsst.multi:201 [INFO] cancelling 17010069 on provider astro proc_lsst.multi INFO: cancelling 17010069 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-13 10:00:56 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-13 10:00:56 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17010111', '17010112', '17010113', '17010114', '17010115', '17010116', '17010117', '17010118', '17010119', '17010120', '17010122', '17010123', '17010124', '17010125', '17010126', '17010127', '17010128', '17010129', '17010130', '17010131', '17010132', '17010133', '17010134', '17010135', '17010136', '17010137', '17010139', '17010140', '17010141', '17010142', '17010143', '17010144', '17010145', '17010146', '17010148', '17010149', '17010150', '17010046', '17010047', '17010048', '17010049', '17010050', '17010051', '17010052', '17010053', '17010054', '17010055', '17010056', '17010057', '17010058', '17010059', '17010060', '17010061', '17010062', '17010063', '17010064', '17010065', '17010066', '17010067', '17010069', '17010072', '17010074', '17010075', '17010076', '17010077', '17010078', '17010080', '17010088', '17010089', '17010091', '17010092', '17010093', '17010094', '17010095', '17010096', '17010097', '17010098', '17010099', '17010100', '17010101', '17010102', '17010104', '17010105', '17010106', '17010107', '17010109', '17010110', '17010081', '17010082', '17010084', '17010086', '17010087', '60730'] proc_lsst.multi INFO: new jobs since last cancel ['17010111', '17010112', '17010113', '17010114', '17010115', '17010116', '17010117', '17010118', '17010119', '17010120', '17010122', '17010123', '17010124', '17010125', '17010126', '17010127', '17010128', '17010129', '17010130', '17010131', '17010132', '17010133', '17010134', '17010135', '17010136', '17010137', '17010139', '17010140', '17010141', '17010142', '17010143', '17010144', '17010145', '17010146', '17010148', '17010149', '17010150', '17010046', '17010047', '17010048', '17010049', '17010050', '17010051', '17010052', '17010053', '17010054', '17010055', '17010056', '17010057', '17010058', '17010059', '17010060', '17010061', '17010062', '17010063', '17010064', '17010065', '17010066', '17010067', '17010069', '17010072', '17010074', '17010075', '17010076', '17010077', '17010078', '17010080', '17010088', '17010089', '17010091', '17010092', '17010093', '17010094', '17010095', '17010096', '17010097', '17010098', '17010099', '17010100', '17010101', '17010102', '17010104', '17010105', '17010106', '17010107', '17010109', '17010110', '17010081', '17010082', '17010084', '17010086', '17010087', '60730'] 2024-03-13 10:00:56 proc_lsst.multi:146 [INFO] found job 17010111 in provider checkpoint proc_lsst.multi INFO: found job 17010111 in provider checkpoint 2024-03-13 10:00:56 proc_lsst.multi:201 [INFO] cancelling 17010111 on provider checkpoint proc_lsst.multi INFO: cancelling 17010111 on provider checkpoint 2024-03-13 10:00:56 proc_lsst.multi:146 [INFO] found job 17010112 in provider checkpoint proc_lsst.multi INFO: found job 17010112 in provider checkpoint 2024-03-13 10:00:56 proc_lsst.multi:201 [INFO] cancelling 17010112 on provider checkpoint proc_lsst.multi INFO: cancelling 17010112 on provider checkpoint 2024-03-13 10:00:56 proc_lsst.multi:146 [INFO] found job 17010113 in provider checkpoint proc_lsst.multi INFO: found job 17010113 in provider checkpoint 2024-03-13 10:00:56 proc_lsst.multi:201 [INFO] cancelling 17010113 on provider checkpoint proc_lsst.multi INFO: cancelling 17010113 on provider checkpoint 2024-03-13 10:00:56 proc_lsst.multi:146 [INFO] found job 17010114 in provider checkpoint proc_lsst.multi INFO: found job 17010114 in provider checkpoint 2024-03-13 10:00:56 proc_lsst.multi:201 [INFO] cancelling 17010114 on provider checkpoint proc_lsst.multi INFO: cancelling 17010114 on provider checkpoint 2024-03-13 10:00:56 proc_lsst.multi:146 [INFO] found job 17010115 in provider checkpoint proc_lsst.multi INFO: found job 17010115 in provider checkpoint 2024-03-13 10:00:56 proc_lsst.multi:201 [INFO] cancelling 17010115 on provider checkpoint proc_lsst.multi INFO: cancelling 17010115 on provider checkpoint 2024-03-13 10:00:56 proc_lsst.multi:146 [INFO] found job 17010116 in provider checkpoint proc_lsst.multi INFO: found job 17010116 in provider checkpoint 2024-03-13 10:00:56 proc_lsst.multi:201 [INFO] cancelling 17010116 on provider checkpoint proc_lsst.multi INFO: cancelling 17010116 on provider checkpoint 2024-03-13 10:00:56 proc_lsst.multi:146 [INFO] found job 17010117 in provider checkpoint proc_lsst.multi INFO: found job 17010117 in provider checkpoint 2024-03-13 10:00:56 proc_lsst.multi:201 [INFO] cancelling 17010117 on provider checkpoint proc_lsst.multi INFO: cancelling 17010117 on provider checkpoint 2024-03-13 10:00:57 proc_lsst.multi:146 [INFO] found job 17010118 in provider checkpoint proc_lsst.multi INFO: found job 17010118 in provider checkpoint 2024-03-13 10:00:57 proc_lsst.multi:201 [INFO] cancelling 17010118 on provider checkpoint proc_lsst.multi INFO: cancelling 17010118 on provider checkpoint 2024-03-13 10:00:57 proc_lsst.multi:146 [INFO] found job 17010119 in provider checkpoint proc_lsst.multi INFO: found job 17010119 in provider checkpoint 2024-03-13 10:00:57 proc_lsst.multi:201 [INFO] cancelling 17010119 on provider checkpoint proc_lsst.multi INFO: cancelling 17010119 on provider checkpoint 2024-03-13 10:00:57 proc_lsst.multi:146 [INFO] found job 17010120 in provider checkpoint proc_lsst.multi INFO: found job 17010120 in provider checkpoint 2024-03-13 10:00:57 proc_lsst.multi:201 [INFO] cancelling 17010120 on provider checkpoint proc_lsst.multi INFO: cancelling 17010120 on provider checkpoint 2024-03-13 10:00:57 proc_lsst.multi:146 [INFO] found job 17010122 in provider checkpoint proc_lsst.multi INFO: found job 17010122 in provider checkpoint 2024-03-13 10:00:57 proc_lsst.multi:201 [INFO] cancelling 17010122 on provider checkpoint proc_lsst.multi INFO: cancelling 17010122 on provider checkpoint 2024-03-13 10:00:57 proc_lsst.multi:146 [INFO] found job 17010123 in provider checkpoint proc_lsst.multi INFO: found job 17010123 in provider checkpoint 2024-03-13 10:00:57 proc_lsst.multi:201 [INFO] cancelling 17010123 on provider checkpoint proc_lsst.multi INFO: cancelling 17010123 on provider checkpoint 2024-03-13 10:00:57 proc_lsst.multi:146 [INFO] found job 17010124 in provider checkpoint proc_lsst.multi INFO: found job 17010124 in provider checkpoint 2024-03-13 10:00:57 proc_lsst.multi:201 [INFO] cancelling 17010124 on provider checkpoint proc_lsst.multi INFO: cancelling 17010124 on provider checkpoint 2024-03-13 10:00:57 proc_lsst.multi:146 [INFO] found job 17010125 in provider checkpoint proc_lsst.multi INFO: found job 17010125 in provider checkpoint 2024-03-13 10:00:57 proc_lsst.multi:201 [INFO] cancelling 17010125 on provider checkpoint proc_lsst.multi INFO: cancelling 17010125 on provider checkpoint 2024-03-13 10:00:57 proc_lsst.multi:146 [INFO] found job 17010126 in provider checkpoint proc_lsst.multi INFO: found job 17010126 in provider checkpoint 2024-03-13 10:00:57 proc_lsst.multi:201 [INFO] cancelling 17010126 on provider checkpoint proc_lsst.multi INFO: cancelling 17010126 on provider checkpoint 2024-03-13 10:00:57 proc_lsst.multi:146 [INFO] found job 17010127 in provider checkpoint proc_lsst.multi INFO: found job 17010127 in provider checkpoint 2024-03-13 10:00:57 proc_lsst.multi:201 [INFO] cancelling 17010127 on provider checkpoint proc_lsst.multi INFO: cancelling 17010127 on provider checkpoint 2024-03-13 10:00:57 proc_lsst.multi:146 [INFO] found job 17010128 in provider checkpoint proc_lsst.multi INFO: found job 17010128 in provider checkpoint 2024-03-13 10:00:57 proc_lsst.multi:201 [INFO] cancelling 17010128 on provider checkpoint proc_lsst.multi INFO: cancelling 17010128 on provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:146 [INFO] found job 17010129 in provider checkpoint proc_lsst.multi INFO: found job 17010129 in provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:201 [INFO] cancelling 17010129 on provider checkpoint proc_lsst.multi INFO: cancelling 17010129 on provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:146 [INFO] found job 17010130 in provider checkpoint proc_lsst.multi INFO: found job 17010130 in provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:201 [INFO] cancelling 17010130 on provider checkpoint proc_lsst.multi INFO: cancelling 17010130 on provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:146 [INFO] found job 17010131 in provider checkpoint proc_lsst.multi INFO: found job 17010131 in provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:201 [INFO] cancelling 17010131 on provider checkpoint proc_lsst.multi INFO: cancelling 17010131 on provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:146 [INFO] found job 17010132 in provider checkpoint proc_lsst.multi INFO: found job 17010132 in provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:201 [INFO] cancelling 17010132 on provider checkpoint proc_lsst.multi INFO: cancelling 17010132 on provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:146 [INFO] found job 17010133 in provider checkpoint proc_lsst.multi INFO: found job 17010133 in provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:201 [INFO] cancelling 17010133 on provider checkpoint proc_lsst.multi INFO: cancelling 17010133 on provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:146 [INFO] found job 17010134 in provider checkpoint proc_lsst.multi INFO: found job 17010134 in provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:201 [INFO] cancelling 17010134 on provider checkpoint proc_lsst.multi INFO: cancelling 17010134 on provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:146 [INFO] found job 17010135 in provider checkpoint proc_lsst.multi INFO: found job 17010135 in provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:201 [INFO] cancelling 17010135 on provider checkpoint proc_lsst.multi INFO: cancelling 17010135 on provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:146 [INFO] found job 17010136 in provider checkpoint proc_lsst.multi INFO: found job 17010136 in provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:201 [INFO] cancelling 17010136 on provider checkpoint proc_lsst.multi INFO: cancelling 17010136 on provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:146 [INFO] found job 17010137 in provider checkpoint proc_lsst.multi INFO: found job 17010137 in provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:201 [INFO] cancelling 17010137 on provider checkpoint proc_lsst.multi INFO: cancelling 17010137 on provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:146 [INFO] found job 17010139 in provider checkpoint proc_lsst.multi INFO: found job 17010139 in provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:201 [INFO] cancelling 17010139 on provider checkpoint proc_lsst.multi INFO: cancelling 17010139 on provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:146 [INFO] found job 17010140 in provider checkpoint proc_lsst.multi INFO: found job 17010140 in provider checkpoint 2024-03-13 10:00:58 proc_lsst.multi:201 [INFO] cancelling 17010140 on provider checkpoint proc_lsst.multi INFO: cancelling 17010140 on provider checkpoint 2024-03-13 10:00:59 proc_lsst.multi:146 [INFO] found job 17010141 in provider checkpoint proc_lsst.multi INFO: found job 17010141 in provider checkpoint 2024-03-13 10:00:59 proc_lsst.multi:201 [INFO] cancelling 17010141 on provider checkpoint proc_lsst.multi INFO: cancelling 17010141 on provider checkpoint 2024-03-13 10:00:59 proc_lsst.multi:146 [INFO] found job 17010142 in provider checkpoint proc_lsst.multi INFO: found job 17010142 in provider checkpoint 2024-03-13 10:00:59 proc_lsst.multi:201 [INFO] cancelling 17010142 on provider checkpoint proc_lsst.multi INFO: cancelling 17010142 on provider checkpoint 2024-03-13 10:00:59 proc_lsst.multi:146 [INFO] found job 17010143 in provider checkpoint proc_lsst.multi INFO: found job 17010143 in provider checkpoint 2024-03-13 10:00:59 proc_lsst.multi:201 [INFO] cancelling 17010143 on provider checkpoint proc_lsst.multi INFO: cancelling 17010143 on provider checkpoint 2024-03-13 10:00:59 proc_lsst.multi:146 [INFO] found job 17010144 in provider checkpoint proc_lsst.multi INFO: found job 17010144 in provider checkpoint 2024-03-13 10:00:59 proc_lsst.multi:201 [INFO] cancelling 17010144 on provider checkpoint proc_lsst.multi INFO: cancelling 17010144 on provider checkpoint 2024-03-13 10:00:59 proc_lsst.multi:146 [INFO] found job 17010145 in provider checkpoint proc_lsst.multi INFO: found job 17010145 in provider checkpoint 2024-03-13 10:00:59 proc_lsst.multi:201 [INFO] cancelling 17010145 on provider checkpoint proc_lsst.multi INFO: cancelling 17010145 on provider checkpoint 2024-03-13 10:00:59 proc_lsst.multi:146 [INFO] found job 17010146 in provider checkpoint proc_lsst.multi INFO: found job 17010146 in provider checkpoint 2024-03-13 10:00:59 proc_lsst.multi:201 [INFO] cancelling 17010146 on provider checkpoint proc_lsst.multi INFO: cancelling 17010146 on provider checkpoint 2024-03-13 10:00:59 proc_lsst.multi:146 [INFO] found job 17010148 in provider checkpoint proc_lsst.multi INFO: found job 17010148 in provider checkpoint 2024-03-13 10:00:59 proc_lsst.multi:201 [INFO] cancelling 17010148 on provider checkpoint proc_lsst.multi INFO: cancelling 17010148 on provider checkpoint 2024-03-13 10:00:59 proc_lsst.multi:146 [INFO] found job 17010149 in provider checkpoint proc_lsst.multi INFO: found job 17010149 in provider checkpoint 2024-03-13 10:00:59 proc_lsst.multi:201 [INFO] cancelling 17010149 on provider checkpoint proc_lsst.multi INFO: cancelling 17010149 on provider checkpoint 2024-03-13 10:00:59 proc_lsst.multi:146 [INFO] found job 17010150 in provider checkpoint proc_lsst.multi INFO: found job 17010150 in provider checkpoint 2024-03-13 10:00:59 proc_lsst.multi:201 [INFO] cancelling 17010150 on provider checkpoint proc_lsst.multi INFO: cancelling 17010150 on provider checkpoint 2024-03-13 10:00:59 proc_lsst.multi:146 [INFO] found job 17010046 in provider astro proc_lsst.multi INFO: found job 17010046 in provider astro 2024-03-13 10:00:59 proc_lsst.multi:201 [INFO] cancelling 17010046 on provider astro proc_lsst.multi INFO: cancelling 17010046 on provider astro 2024-03-13 10:00:59 proc_lsst.multi:146 [INFO] found job 17010047 in provider astro proc_lsst.multi INFO: found job 17010047 in provider astro 2024-03-13 10:00:59 proc_lsst.multi:201 [INFO] cancelling 17010047 on provider astro proc_lsst.multi INFO: cancelling 17010047 on provider astro 2024-03-13 10:01:00 proc_lsst.multi:146 [INFO] found job 17010048 in provider astro proc_lsst.multi INFO: found job 17010048 in provider astro 2024-03-13 10:01:00 proc_lsst.multi:201 [INFO] cancelling 17010048 on provider astro proc_lsst.multi INFO: cancelling 17010048 on provider astro 2024-03-13 10:01:00 proc_lsst.multi:146 [INFO] found job 17010049 in provider astro proc_lsst.multi INFO: found job 17010049 in provider astro 2024-03-13 10:01:00 proc_lsst.multi:201 [INFO] cancelling 17010049 on provider astro proc_lsst.multi INFO: cancelling 17010049 on provider astro 2024-03-13 10:01:00 proc_lsst.multi:146 [INFO] found job 17010050 in provider astro proc_lsst.multi INFO: found job 17010050 in provider astro 2024-03-13 10:01:00 proc_lsst.multi:201 [INFO] cancelling 17010050 on provider astro proc_lsst.multi INFO: cancelling 17010050 on provider astro 2024-03-13 10:01:00 proc_lsst.multi:146 [INFO] found job 17010051 in provider astro proc_lsst.multi INFO: found job 17010051 in provider astro 2024-03-13 10:01:00 proc_lsst.multi:201 [INFO] cancelling 17010051 on provider astro proc_lsst.multi INFO: cancelling 17010051 on provider astro 2024-03-13 10:01:00 proc_lsst.multi:146 [INFO] found job 17010052 in provider astro proc_lsst.multi INFO: found job 17010052 in provider astro 2024-03-13 10:01:00 proc_lsst.multi:201 [INFO] cancelling 17010052 on provider astro proc_lsst.multi INFO: cancelling 17010052 on provider astro 2024-03-13 10:01:00 proc_lsst.multi:146 [INFO] found job 17010053 in provider astro proc_lsst.multi INFO: found job 17010053 in provider astro 2024-03-13 10:01:00 proc_lsst.multi:201 [INFO] cancelling 17010053 on provider astro proc_lsst.multi INFO: cancelling 17010053 on provider astro 2024-03-13 10:01:00 proc_lsst.multi:146 [INFO] found job 17010054 in provider astro proc_lsst.multi INFO: found job 17010054 in provider astro 2024-03-13 10:01:00 proc_lsst.multi:201 [INFO] cancelling 17010054 on provider astro proc_lsst.multi INFO: cancelling 17010054 on provider astro 2024-03-13 10:01:00 proc_lsst.multi:146 [INFO] found job 17010055 in provider astro proc_lsst.multi INFO: found job 17010055 in provider astro 2024-03-13 10:01:00 proc_lsst.multi:201 [INFO] cancelling 17010055 on provider astro proc_lsst.multi INFO: cancelling 17010055 on provider astro 2024-03-13 10:01:00 proc_lsst.multi:146 [INFO] found job 17010056 in provider astro proc_lsst.multi INFO: found job 17010056 in provider astro 2024-03-13 10:01:00 proc_lsst.multi:201 [INFO] cancelling 17010056 on provider astro proc_lsst.multi INFO: cancelling 17010056 on provider astro 2024-03-13 10:01:00 proc_lsst.multi:146 [INFO] found job 17010057 in provider astro proc_lsst.multi INFO: found job 17010057 in provider astro 2024-03-13 10:01:00 proc_lsst.multi:201 [INFO] cancelling 17010057 on provider astro proc_lsst.multi INFO: cancelling 17010057 on provider astro 2024-03-13 10:01:00 proc_lsst.multi:146 [INFO] found job 17010058 in provider astro proc_lsst.multi INFO: found job 17010058 in provider astro 2024-03-13 10:01:00 proc_lsst.multi:201 [INFO] cancelling 17010058 on provider astro proc_lsst.multi INFO: cancelling 17010058 on provider astro 2024-03-13 10:01:01 proc_lsst.multi:146 [INFO] found job 17010059 in provider astro proc_lsst.multi INFO: found job 17010059 in provider astro 2024-03-13 10:01:01 proc_lsst.multi:201 [INFO] cancelling 17010059 on provider astro proc_lsst.multi INFO: cancelling 17010059 on provider astro 2024-03-13 10:01:01 proc_lsst.multi:146 [INFO] found job 17010060 in provider astro proc_lsst.multi INFO: found job 17010060 in provider astro 2024-03-13 10:01:01 proc_lsst.multi:201 [INFO] cancelling 17010060 on provider astro proc_lsst.multi INFO: cancelling 17010060 on provider astro 2024-03-13 10:01:01 proc_lsst.multi:146 [INFO] found job 17010061 in provider astro proc_lsst.multi INFO: found job 17010061 in provider astro 2024-03-13 10:01:01 proc_lsst.multi:201 [INFO] cancelling 17010061 on provider astro proc_lsst.multi INFO: cancelling 17010061 on provider astro 2024-03-13 10:01:01 proc_lsst.multi:146 [INFO] found job 17010062 in provider astro proc_lsst.multi INFO: found job 17010062 in provider astro 2024-03-13 10:01:01 proc_lsst.multi:201 [INFO] cancelling 17010062 on provider astro proc_lsst.multi INFO: cancelling 17010062 on provider astro 2024-03-13 10:01:01 proc_lsst.multi:146 [INFO] found job 17010063 in provider astro proc_lsst.multi INFO: found job 17010063 in provider astro 2024-03-13 10:01:01 proc_lsst.multi:201 [INFO] cancelling 17010063 on provider astro proc_lsst.multi INFO: cancelling 17010063 on provider astro 2024-03-13 10:01:01 proc_lsst.multi:146 [INFO] found job 17010064 in provider astro proc_lsst.multi INFO: found job 17010064 in provider astro 2024-03-13 10:01:01 proc_lsst.multi:201 [INFO] cancelling 17010064 on provider astro proc_lsst.multi INFO: cancelling 17010064 on provider astro 2024-03-13 10:01:01 proc_lsst.multi:146 [INFO] found job 17010065 in provider astro proc_lsst.multi INFO: found job 17010065 in provider astro 2024-03-13 10:01:01 proc_lsst.multi:201 [INFO] cancelling 17010065 on provider astro proc_lsst.multi INFO: cancelling 17010065 on provider astro 2024-03-13 10:01:01 proc_lsst.multi:146 [INFO] found job 17010066 in provider astro proc_lsst.multi INFO: found job 17010066 in provider astro 2024-03-13 10:01:01 proc_lsst.multi:201 [INFO] cancelling 17010066 on provider astro proc_lsst.multi INFO: cancelling 17010066 on provider astro 2024-03-13 10:01:01 proc_lsst.multi:146 [INFO] found job 17010067 in provider astro proc_lsst.multi INFO: found job 17010067 in provider astro 2024-03-13 10:01:01 proc_lsst.multi:201 [INFO] cancelling 17010067 on provider astro proc_lsst.multi INFO: cancelling 17010067 on provider astro 2024-03-13 10:01:01 proc_lsst.multi:146 [INFO] found job 17010069 in provider astro proc_lsst.multi INFO: found job 17010069 in provider astro 2024-03-13 10:01:01 proc_lsst.multi:201 [INFO] cancelling 17010069 on provider astro proc_lsst.multi INFO: cancelling 17010069 on provider astro 2024-03-13 10:01:02 proc_lsst.multi:146 [INFO] found job 17010072 in provider astro proc_lsst.multi INFO: found job 17010072 in provider astro 2024-03-13 10:01:02 proc_lsst.multi:201 [INFO] cancelling 17010072 on provider astro proc_lsst.multi INFO: cancelling 17010072 on provider astro 2024-03-13 10:01:02 proc_lsst.multi:146 [INFO] found job 17010074 in provider astro proc_lsst.multi INFO: found job 17010074 in provider astro 2024-03-13 10:01:02 proc_lsst.multi:201 [INFO] cancelling 17010074 on provider astro proc_lsst.multi INFO: cancelling 17010074 on provider astro 2024-03-13 10:01:02 proc_lsst.multi:146 [INFO] found job 17010075 in provider astro proc_lsst.multi INFO: found job 17010075 in provider astro 2024-03-13 10:01:02 proc_lsst.multi:201 [INFO] cancelling 17010075 on provider astro proc_lsst.multi INFO: cancelling 17010075 on provider astro 2024-03-13 10:01:02 proc_lsst.multi:146 [INFO] found job 17010076 in provider astro proc_lsst.multi INFO: found job 17010076 in provider astro 2024-03-13 10:01:02 proc_lsst.multi:201 [INFO] cancelling 17010076 on provider astro proc_lsst.multi INFO: cancelling 17010076 on provider astro 2024-03-13 10:01:02 proc_lsst.multi:146 [INFO] found job 17010077 in provider astro proc_lsst.multi INFO: found job 17010077 in provider astro 2024-03-13 10:01:02 proc_lsst.multi:201 [INFO] cancelling 17010077 on provider astro proc_lsst.multi INFO: cancelling 17010077 on provider astro 2024-03-13 10:01:02 proc_lsst.multi:146 [INFO] found job 17010078 in provider astro proc_lsst.multi INFO: found job 17010078 in provider astro 2024-03-13 10:01:02 proc_lsst.multi:201 [INFO] cancelling 17010078 on provider astro proc_lsst.multi INFO: cancelling 17010078 on provider astro 2024-03-13 10:01:02 proc_lsst.multi:146 [INFO] found job 17010080 in provider astro proc_lsst.multi INFO: found job 17010080 in provider astro 2024-03-13 10:01:02 proc_lsst.multi:201 [INFO] cancelling 17010080 on provider astro proc_lsst.multi INFO: cancelling 17010080 on provider astro 2024-03-13 10:01:02 proc_lsst.multi:146 [INFO] found job 17010088 in provider escience_a40 proc_lsst.multi INFO: found job 17010088 in provider escience_a40 2024-03-13 10:01:02 proc_lsst.multi:201 [INFO] cancelling 17010088 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010088 on provider escience_a40 2024-03-13 10:01:02 proc_lsst.multi:146 [INFO] found job 17010089 in provider escience_a40 proc_lsst.multi INFO: found job 17010089 in provider escience_a40 2024-03-13 10:01:02 proc_lsst.multi:201 [INFO] cancelling 17010089 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010089 on provider escience_a40 2024-03-13 10:01:02 proc_lsst.multi:146 [INFO] found job 17010091 in provider escience_a40 proc_lsst.multi INFO: found job 17010091 in provider escience_a40 2024-03-13 10:01:02 proc_lsst.multi:201 [INFO] cancelling 17010091 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010091 on provider escience_a40 2024-03-13 10:01:02 proc_lsst.multi:146 [INFO] found job 17010092 in provider escience_a40 proc_lsst.multi INFO: found job 17010092 in provider escience_a40 2024-03-13 10:01:02 proc_lsst.multi:201 [INFO] cancelling 17010092 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010092 on provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:146 [INFO] found job 17010093 in provider escience_a40 proc_lsst.multi INFO: found job 17010093 in provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:201 [INFO] cancelling 17010093 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010093 on provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:146 [INFO] found job 17010094 in provider escience_a40 proc_lsst.multi INFO: found job 17010094 in provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:201 [INFO] cancelling 17010094 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010094 on provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:146 [INFO] found job 17010095 in provider escience_a40 proc_lsst.multi INFO: found job 17010095 in provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:201 [INFO] cancelling 17010095 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010095 on provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:146 [INFO] found job 17010096 in provider escience_a40 proc_lsst.multi INFO: found job 17010096 in provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:201 [INFO] cancelling 17010096 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010096 on provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:146 [INFO] found job 17010097 in provider escience_a40 proc_lsst.multi INFO: found job 17010097 in provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:201 [INFO] cancelling 17010097 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010097 on provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:146 [INFO] found job 17010098 in provider escience_a40 proc_lsst.multi INFO: found job 17010098 in provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:201 [INFO] cancelling 17010098 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010098 on provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:146 [INFO] found job 17010099 in provider escience_a40 proc_lsst.multi INFO: found job 17010099 in provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:201 [INFO] cancelling 17010099 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010099 on provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:146 [INFO] found job 17010100 in provider escience_a40 proc_lsst.multi INFO: found job 17010100 in provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:201 [INFO] cancelling 17010100 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010100 on provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:146 [INFO] found job 17010101 in provider escience_a40 proc_lsst.multi INFO: found job 17010101 in provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:201 [INFO] cancelling 17010101 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010101 on provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:146 [INFO] found job 17010102 in provider escience_a40 proc_lsst.multi INFO: found job 17010102 in provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:201 [INFO] cancelling 17010102 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010102 on provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:146 [INFO] found job 17010104 in provider escience_a40 proc_lsst.multi INFO: found job 17010104 in provider escience_a40 2024-03-13 10:01:03 proc_lsst.multi:201 [INFO] cancelling 17010104 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010104 on provider escience_a40 2024-03-13 10:01:04 proc_lsst.multi:146 [INFO] found job 17010105 in provider escience_a40 proc_lsst.multi INFO: found job 17010105 in provider escience_a40 2024-03-13 10:01:04 proc_lsst.multi:201 [INFO] cancelling 17010105 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010105 on provider escience_a40 2024-03-13 10:01:04 proc_lsst.multi:146 [INFO] found job 17010106 in provider escience_a40 proc_lsst.multi INFO: found job 17010106 in provider escience_a40 2024-03-13 10:01:04 proc_lsst.multi:201 [INFO] cancelling 17010106 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010106 on provider escience_a40 2024-03-13 10:01:04 proc_lsst.multi:146 [INFO] found job 17010107 in provider escience_a40 proc_lsst.multi INFO: found job 17010107 in provider escience_a40 2024-03-13 10:01:04 proc_lsst.multi:201 [INFO] cancelling 17010107 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010107 on provider escience_a40 2024-03-13 10:01:04 proc_lsst.multi:146 [INFO] found job 17010109 in provider escience_a40 proc_lsst.multi INFO: found job 17010109 in provider escience_a40 2024-03-13 10:01:04 proc_lsst.multi:201 [INFO] cancelling 17010109 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010109 on provider escience_a40 2024-03-13 10:01:04 proc_lsst.multi:146 [INFO] found job 17010110 in provider escience_a40 proc_lsst.multi INFO: found job 17010110 in provider escience_a40 2024-03-13 10:01:04 proc_lsst.multi:201 [INFO] cancelling 17010110 on provider escience_a40 proc_lsst.multi INFO: cancelling 17010110 on provider escience_a40 2024-03-13 10:01:04 proc_lsst.multi:146 [INFO] found job 17010081 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010081 in provider escience_rtx6k 2024-03-13 10:01:04 proc_lsst.multi:201 [INFO] cancelling 17010081 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17010081 on provider escience_rtx6k 2024-03-13 10:01:04 proc_lsst.multi:146 [INFO] found job 17010082 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010082 in provider escience_rtx6k 2024-03-13 10:01:04 proc_lsst.multi:201 [INFO] cancelling 17010082 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17010082 on provider escience_rtx6k 2024-03-13 10:01:04 proc_lsst.multi:146 [INFO] found job 17010084 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010084 in provider escience_rtx6k 2024-03-13 10:01:04 proc_lsst.multi:201 [INFO] cancelling 17010084 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17010084 on provider escience_rtx6k 2024-03-13 10:01:04 proc_lsst.multi:146 [INFO] found job 17010086 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010086 in provider escience_rtx6k 2024-03-13 10:01:04 proc_lsst.multi:201 [INFO] cancelling 17010086 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17010086 on provider escience_rtx6k 2024-03-13 10:01:04 proc_lsst.multi:146 [INFO] found job 17010087 in provider escience_rtx6k proc_lsst.multi INFO: found job 17010087 in provider escience_rtx6k 2024-03-13 10:01:04 proc_lsst.multi:201 [INFO] cancelling 17010087 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17010087 on provider escience_rtx6k 2024-03-13 10:01:04 proc_lsst.multi:146 [INFO] found job 60730 in provider local proc_lsst.multi INFO: found job 60730 in provider local 2024-03-13 10:01:04 proc_lsst.multi:201 [INFO] cancelling 60730 on provider local proc_lsst.multi INFO: cancelling 60730 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 60730 parsl.providers.local.local WARNING: Failed to kill PID: 60730 and child processes on local 2024-03-13 10:01:06 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 986.7193 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20190927_B1a_science#step1_20240313T094419Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 986.7256 seconds; current memory usage: 0.387 Gibyte, delta: 0.009 Gibyte, peak delta: 0.009 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 1006.0719 seconds; current memory usage: 0.387 Gibyte, delta: 0.251 Gibyte, peak delta: 0.251 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.387 Gibyte (main), 0.387 Gibyte (largest child process) Run Id: None Run Name: DEEP_20190927_B1a_science#step1_20240313T094419Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 16m48.433s user 3m24.668s sys 2m11.570s